loading...
Comment author: Yvain 01 May 2016 05:11:57PM 19 points [-]

Nice work.

If possible, please do a formal writeup like this: http://lesswrong.com/lw/lhg/2014_survey_results/

If possible, please change the data on your PDF file to include an option to have it without nonresponders. For example, right now sex is 66% male, 12% female, unknown 22%, which makes it hard to intuitively tell what the actual sex ratio is. If you remove the unknowns you see that the knowns are 85% male 15% female, which is a much more useful result. This is especially true since up to 50% of people are unknowns on some questions.

If possible, please include averages for numerical questions. For example, there's no data about age on the PDF file because it just says everybody was a "responder" but doesn't list numbers.

Comment author: ingres 26 March 2016 02:46:40AM *  3 points [-]

Yeah, you're right.

Currently trying to figure out how to do that in the least intrusive way.

EDIT: Good news it turns out that I can edit the calibration question 'answers' after all. The ones where a range would make sense have been edited to include one. Questions such as "which is heavier" have not been because the ignorance prior should be fairly obvious.

Fri Mar 25 19:50:41 PDT 2016 | Answers on or before this date where the ranges have been added will be controlled for at analysis time.

Comment author: Yvain 26 March 2016 02:49:37AM 1 point [-]

If you throw out the data, I request you keep the thrown-out data somewhere else so I can see how people responded to the issue.

In response to comment by Elo on Lesswrong 2016 Survey
Comment author: ingres 26 March 2016 02:27:17AM *  1 point [-]

We will get that suggestion sorted asap.

I actually can't do that. The way our survey engine works changing the question answers mid-survey would require taking it down for maintenance and hand-joining the current respondents to the new respondents. In general I planned to handle the "within 10 cm" thing during analysis. Try to fermi estimate the value and give your closest answer, then the probability you got it right. We can look at how close your confidence was to a sane range of values for the answer.

I.E, if you got it within ten and said you had a ten percent chance of getting it right you're well calibrated.

Note: I am not entirely sure this is sane, and would like feedback on better ways to do it.

EDIT: I should probably be very precise here. I cannot change the question answers in the software, presumably because it would involve changing the underlying table schema for the database. I can change the question/ question descriptions so if there's a superior process for answering these I could describe it there.

Comment author: Yvain 26 March 2016 02:42:51AM *  6 points [-]

"In general I planned to handle the "within 10 cm" thing during analysis. Try to fermi estimate the value and give your closest answer, then the probability you got it right. We can look at how close your confidence was to a sane range of values for the answer."

But unless I'm misunderstanding you, the size of the unspoken "sane range" is the entire determinant of how you should calibrate yourself.

Suppose you ask me when Genghis Khan was born, and all I know is "sometime between 1100 and 1200, with certainty". Suppose I choose 1150. If you require the exact year, then I'm only right if it was exactly 1150, and since it could be any of 100 years my probability is 1%. If you require within five years, then I'm right if it was any time between 1145 and 1155, so my probability is 10%. If you require within fifty years, then my probability is effectively 100%. All of those are potential "sane ranges", but depending on which one you the correctly calibrated estimate could be anywhere from 1% to 100%.

Unless I am very confused, you might want to change the questions and hand-throw-out all the answers you received before now, since I don't think they're meaningful (except if interpreted as probability of being exactly right).

(Actually, it might be interesting to see how many people figure this out, in a train wreck sort of way.)

PS: I admit this is totally 100% my fault for not getting around to looking at it the five times you asked me to before this.

Comment author: Yvain 26 March 2016 02:04:36AM 24 points [-]

Elo, thanks a lot for doing this.

(for the record, Elo tried really hard to get me involved and I procrastinated helping and forgot about it. I 100% endorse this.)

My only suggestion is to create a margin of error on the calibration questions, eg "How big is the soccer ball, to within 10 cm?". Otherwise people are guessing whether they got the exact centimeter right, which is pretty hard.

Comment author: Lumifer 15 March 2016 04:49:41PM *  10 points [-]

This post by Eric Raymond should be interesting to LW :-) Extended quoting:

There’s a link between autism and genius says a popular-press summary of recent research. If you follow this sort of thing (and I do) most of what follows doesn’t come as much of a surprise. We get the usual thumbnail case studies about autistic savants. There’s an interesting thread about how child prodigies who are not autists rely on autism-like facilities for pattern recognition and hyperconcentration. There’s a sketch of research suggesting that non-autistic child-prodigies, like autists, tend to have exceptionally large working memories. Often, they have autistic relatives. Money quote: “Recent study led by a University of Edinburgh researcher found that in non-autistic adults, having more autism-linked genetic variants was associated with better cognitive function.”

But then I got to this: “In a way, this link to autism only deepens the prodigy mystery.” And my instant reaction was: “Mystery? There’s a mystery here? What?” Rereading, it seems that the authors (and other researchers) are mystified by the question of exactly how autism-like traits promote genius-level capabilities.

At which point I blinked and thought: “Eh? It’s right in front of you! How obvious does it have to get before you’ll see it?”

... Yes, there is an enabling superpower that autists have through damage and accident, but non-autists like me have to cultivate: not giving a shit about monkey social rituals.

Neurotypicals spend most of their cognitive bandwidth on mutual grooming and status-maintainance activity. They have great difficulty sustaining interest in anything that won’t yield a near-immediate social reward. By an autist’s standards (or mine) they’re almost always running in a hamster wheel as fast as they can, not getting anywhere.

The neurotypical human mind is designed to compete at this monkey status grind and has zero or only a vanishingly small amount of bandwidth to spare for anything else. Autists escape this trap by lacking the circuitry required to fully solve the other-minds problem; thus, even if their total processing capacity is average or subnormal, they have a lot more of it to spend on what neurotypicals interpret as weird savant talents.

Non-autists have it tougher. To do the genius thing, they have to be either so bright that they can do the monkey status grind with a tiny fraction of their cognitive capability, or train themselves into indifference so they basically don’t care if they lose the neurotypical social game.

Once you realize this it’s easy to understand why the incidence of socially-inept nerdiness doesn’t peak at the extreme high end of the IQ bell curve, but rather in the gifted-to-low-end-genius region closer to the median. I had my nose memorably rubbed in this one time when I was a guest speaker at the Institute for Advanced Study. Afternoon tea was not a nerdfest; it was a roomful of people who are good at the social game because they are good at just about anything they choose to pay attention to and the monkey status grind just isn’t very difficult. Not compared to, say, solving tensor equations.

Comment author: Yvain 18 March 2016 08:51:25PM 1 point [-]

This idea of having more "bandwidth" is tempting, but not really scientifically supported as far as I can tell, unless he just means autists have more free time/energy than neurotypicals.

Comment author: Yvain 18 March 2016 08:46:44PM *  7 points [-]

If race were a factor in twin studies, I think it would show up only in shared environment, since it differs between families but never within families (and is not differently different in MZ vs. DZ twins). That means it would not show in "heredity", unless we're talking about interracial couples with two children, each of whom by coincidence got a very different number of genes from the parents' two races - I think this is rare enough not to matter in real life studies.

Your point stands about the general role of these kinds of things, I just don't think it's counted that way in the twin studies we actually have.

You're right about beauty etc, though. Genetic studies are most informative about interventions to change individuals' standings relative to other individuals, not about interventions to completely change the nature of the playing field.

Comment author: Yvain 17 September 2015 05:33:50AM *  14 points [-]

I don't know if this solves very much. As you say, if we use the number 1, then we shouldn't wear seatbelts, get fire insurance, or eat healthy to avoid getting cancer, since all of those can be classified as Pascal's Muggings. But if we start going for less than one, then we're just defining away Pascal's Mugging by fiat, saying "this is the level at which I am willing to stop worrying about this".

Also, as some people elsewhere in the comments have pointed out, this makes probability non-additive in an awkward sort of way. Suppose that if you eat unhealthy, you increase your risk of one million different diseases by plus one-in-a-million chance of getting each. Suppose also that eating healthy is a mildly unpleasant sacrifice, but getting a disease is much worse. If we calculate this out disease-by-disease, each disease is a Pascal's Mugging and we should choose to eat unhealthy. But if we calculate this out in the broad category of "getting some disease or other", then our chances are quite high and we should eat healthy. But it's very strange that our ontology/categorization scheme should affect our decision-making. This becomes much more dangerous when we start talking about AIs.

Also, does this create weird nonlinear thresholds? For example, suppose that you live on average 80 years. If some event which causes you near-infinite disutility happens every 80.01 years, you should ignore it; if it happens every 79.99 years, then preventing it becomes the entire focus of your existence. But it seems nonsensical for your behavior to change so drastically based on whether an event is every 79.99 years or every 80.01 years.

Also, a world where people follow this plan is a world where I make a killing on the Inverse Lottery (rules: 10,000 people take tickets; each ticket holder gets paid $1, except a randomly chosen "winner" who must pay $20,000)

Comment author: [deleted] 07 July 2015 01:19:07PM 0 points [-]

Nope! I remembered a meeting happening in Ann Arbor but it appears that was a once-off thing, as I can't find any links to a group based there. Not to mention I couldn't make it there anyhow, anytime soon.

In response to comment by [deleted] on Open Thread, Jul. 6 - Jul. 12, 2015
Comment author: Yvain 16 July 2015 02:09:49PM 1 point [-]

There are meetings in the area every couple of months. There's no specific group to link to, but if you give me your email address I will add you to the list.

If you tell me where exactly in Michigan you are, I can try to put you in touch with other Michigan LW/SSC readers. Most are in Ann Arbor, but there are several in the Detroit metro area and at least one in Grand Rapids.

Comment author: gjm 07 May 2015 10:25:03AM 15 points [-]

There was a discussion a little while back (I think in another open thread) about the game of looking at the titles of articles linked from the "Recent on rationality blogs" sidebar and guessing who wrote them. Usually this is pretty easy.

Right now, though, the top link in the list is "The Future is Filters", which seems like an obvious Robin Hanson title. But no! It's Scott, not Robin, and it's about "filter bubbles" rather than "great filters".

I wonder whether Scott is aware of the game and deliberately trying to tease...

Comment author: Yvain 10 May 2015 05:08:30AM *  3 points [-]

I liked the sound of "The Future Is Pipes" and saved that sentence structure in case I needed it.

Comment author: DanArmak 25 February 2015 09:58:08PM *  6 points [-]

People here should be aware of fresh Word of God. Apparently we're NOT in the Mirror.

Comment author: Yvain 26 February 2015 08:48:19PM *  10 points [-]

The Mirror did not touch the ground; the golden frame had no feet. It didn't look like it was hovering; it looked like it was fixed in place, more solid and more motionless than the walls themselves, like it was nailed to the reference frame of the Earth's motion.

The Mirror is in the fourth wall. Now that we-the-readers have seen the mirror, we have to consider that our seeing Eliezer saying this isn't in the mirror might just be part of our coherent extrapolated volition.

Comment author: linkhyrule5 25 February 2015 07:40:08PM 1 point [-]

Presumably, the body was repaired to true health - which also means that the brain was restored to its original state, that is, functional.

At that point, a Muggle can be revived by simply restarting her heart. A witch, apparently, requires a bit of magic.

Comment author: Yvain 25 February 2015 09:46:24PM 2 points [-]

But that suggests that you can resurrect someone non-permanently without the Stone - and possibly keep them alive indefinitely by expending constant magic on it like Harry does with his father's rock.

Comment author: Yvain 25 February 2015 09:44:40PM 9 points [-]

Why is Voldemort not getting rid of Harry in some more final way?

Even if he's worried killing Harry will rebound against him because of the prophecy somehow, he can, I don't know, freeze Harry? Stick Harry in the mirror using whatever happened to Dumbledore? Destroy Harry's brain and memories and leave him an idiot? Shoot Harry into space?

Why is "resurrect Harry's best friend to give him good counsel" a winning move here?

Comment author: Yvain 25 February 2015 07:26:18PM 4 points [-]

How is Voldemort resurrecting Hermione?

With his own resurrection, he's transfiguring stuff into his body, using the Stone to make the transfiguration permanent, then having his spirit repossess the body.

With Hermione, the body was never the problem. Where's he getting the spirit? How does the Stone help?

Comment author: Yvain 25 February 2015 07:08:27PM *  25 points [-]

So Harry gets his wand back, gets his pouch back, Voldemort resurrects Hermione with superpowers, then Voldemort becomes super-weak, his horcruxes mysteriously stop working, and he mentions this is happening loudly enough for Harry to hear and kill him?

Either this is still all in the mirror, or Harry needs to buy lottery tickets right away.

Comment author: Yvain 24 February 2015 11:01:12PM *  12 points [-]

"Well," said Albus Dumbledore. "I do feel stupid."

"I should hope so," Professor Quirrell said easily; if he had been at all shocked himself at being caught, it did not show. A casual wave of his hand changed his robes back to a Professor's clothing.

Dumbledore's grimness had returned and redoubled. "There I am, searching so hard for Voldemort's shade, never noticing that the Defense Professor of Hogwarts is a sickly, half-dead victim possessed by a spirit far more powerful than himself. I would call it senility, if so many others had not missed it as well."

This is Dumbledore admitting he held the Idiot Ball. We've been promised nobody's holding the Idiot Ball. So something's up. I like the theory that mirror is operating as intended and showing Voldemort what he wants to see, that being Dumbledore making a mistake and losing. Alternately, it could be that Dumbledore was somehow able to make a version of him inside the mirror (maybe in his CEV he saw himself inside the mirror protecting the Stone, and that reflection-Dumbledore gained independent existence?). Or he could just have something else up his sleeve.

Comment author: zedzed 16 February 2015 01:39:48AM 3 points [-]

What are some sensible-sounding alternatives to eliminating the snitch entirely?

The best I can think of is have two snitches—red snitch, blue snitch. Whenever a seeker catches their snitch, the opposing team can't score any more; the game ends when the second snitch is caught.

Comment author: Yvain 16 February 2015 05:22:54PM 24 points [-]

Prediction:

Harry gets the Snitch eliminated from Quidditch. Not just in Hogwarts, but in the big leagues as well - they don't want a Germany vs. Austria on their hands.

All of the celebrity Quidditch players of the world - Victor Krum, Ludo Bagman, Finbar Quigley - are distraught by these sudden and drastic changes to a traditional game they've loved for many years. At the ceremony marking the changes, some of them tear up.

The Daily Prophet headline is "BOY WHO LIVED TEARS UP THE STARS"

Eliezer gives all of us a long lecture about how the prior for somebody making celebrities cry is so much higher than the prior for someone literally ripping the Sun apart that the latter hypothesis should never even have entered our consideration, regardless of how much more natural an interpretation of the prophecy it is.

Comment author: Error 16 January 2015 12:54:20AM 3 points [-]

I'm looking for an old post of Eliezer's. If I remember the post correctly, he was commenting that a lot of the negative reaction to evopsych might come from having first encountered it in the hands of dumb internet commentators, instead of <some scholarly book title>.

I don't remember the title he referenced, and the search function is failing me. Can anyone point me in the right direction?

Comment author: Yvain 16 January 2015 01:30:44AM 4 points [-]

I can't find the exact post you're talking about, but the book involved was probably The Adapted Mind, since Eliezer often praises it in terms like those.

2014 Survey Results

87 Yvain 05 January 2015 07:36PM

Thanks to everyone who took the 2014 Less Wrong Census/Survey. Extra thanks to Ozy, who did a lot of the number crunching work.

This year's results are below. Some of them may make more sense in the context of the original survey questions, which can be seen here. Please do not try to take the survey as it is over and your results will not be counted.

I. Population

There were 1503 respondents over 27 days. The last survey got 1636 people over 40 days. The last four full days of the survey saw nineteen, six, and four responses, for an average of about ten. If we assume the next thirteen days had also gotten an average of ten responses - which is generous, since responses tend to trail off with time - then we would have gotten about as many people as the last survey. There is no good evidence here of a decline in population, although it is perhaps compatible with a very small decline.

II. Demographics

Sex
Female: 179, 11.9%
Male: 1311, 87.2%

Gender
F (cisgender): 150, 10.0%
F (transgender MtF): 24, 1.6%
M (cisgender): 1245, 82.8%
M (transgender FtM): 5, 0.3%
Other: 64, 4.3%

Sexual Orientation
Asexual: 59, 3.9%
Bisexual: 216, 14.4%
Heterosexual: 1133, 75.4%
Homosexual: 47, 3.1%
Other: 35, 2.3%

[This question was poorly worded and should have acknowledged that people can both be asexual and have a specific orientation; as a result it probably vastly undercounted our asexual readers]

Relationship Style
Prefer monogamous: 778, 51.8%
Prefer polyamorous: 227, 15.1%
Uncertain/no preference: 464, 30.9%
Other: 23, 1.5%

Number of Partners
0: 738, 49.1%
1: 674, 44.8%
2: 51, 3.4%
3: 17, 1.1%
4: 7, 0.5%
5: 1, 0.1%
Lots and lots: 3, 0.2%

Relationship Goals
Currently not looking for new partners: 648, 43.1%
Open to new partners: 467, 31.1%
Seeking more partners: 370, 24.6%

[22.2% of people who don’t have a partner aren’t looking for one.]


Relationship Status
Married: 274, 18.2%
Relationship: 424, 28.2%
Single: 788, 52.4%

[6.9% of single people have at least one partner; 1.8% have more than one.]

Living With
Alone: 345, 23.0%
With parents and/or guardians: 303, 20.2%
With partner and/or children: 411, 27.3%
With roommates: 428, 28.5%

Children
0: 1317, 81.6%
1: 66, 4.4%
2: 78, 5.2%
3: 17, 1.1%
4: 6, 0.4%
5: 3, 0.2%
6: 1, 0.1%
Lots and lots: 1, 0.1%

Want More Children?
Yes: 549, 36.1%
Uncertain: 426, 28.3%
No: 516, 34.3%

[418 of the people who don’t have children don’t want any, suggesting that the LW community is 27.8% childfree.]

Country
United States, 822, 54.7%
United Kingdom, 116, 7.7%
Canada, 88, 5.9%
Australia: 83, 5.5%
Germany, 62, 4.1%
Russia, 26, 1.7%
Finland, 20, 1.3%
New Zealand, 20, 1.3%
India, 17, 1.1%
Brazil: 15, 1.0%
France, 15, 1.0%
Israel, 15, 1.0%

Lesswrongers Per Capita
Finland: 1/271,950
New Zealand: 1/223,550
Australia: 1/278,674
United States: 1/358,390
Canada: 1/399,545
Israel: 1/537,266
United Kingdom: 1/552,586
Germany: 1/1,290,323
France: 1/ 4,402,000
Russia: 1/ 5,519,231
Brazil: 1/ 13,360,000
India: 1/ 73,647,058

Race
Asian (East Asian): 59. 3.9%
Asian (Indian subcontinent): 33, 2.2%
Black: 12. 0.8%
Hispanic: 32, 2.1%
Middle Eastern: 9, 0.6%
Other: 50, 3.3%
White (non-Hispanic): 1294, 86.1%

Work Status
Academic (teaching): 86, 5.7%
For-profit work: 492, 32.7%
Government work: 59, 3.9%
Homemaker: 8, 0.5%
Independently wealthy: 9, 0.6%
Nonprofit work: 58, 3.9%
Self-employed: 122, 5.8%
Student: 553, 36.8%
Unemployed: 103, 6.9%

Profession
Art: 22, 1.5%
Biology: 29, 1.9%
Business: 35, 4.0%
Computers (AI): 42, 2.8%
Computers (other academic): 106, 7.1%
Computers (practical): 477, 31.7%
Engineering: 104, 6.1%
Finance/Economics: 71, 4.7%
Law: 38, 2.5%
Mathematics: 121, 8.1%
Medicine: 32, 2.1%
Neuroscience: 18, 1.2%
Philosophy: 36, 2.4%
Physics: 65, 4.3%
Psychology: 31, 2.1%
Other: 157, 10.2%
Other “hard science”: 25, 1.7%
Other “social science”: 34, 2.3%

Degree
None: 74, 4.9%
High school: 347, 23.1%
2 year degree: 64, 4.3%
Bachelors: 555, 36.9%
Masters: 278, 18.5%
JD/MD/other professional degree: 44, 2.9%
PhD: 105, 7.0%
Other: 24, 1.4%

III. Mental Illness

535 answer “no” to all the mental illness questions. Upper bound: 64.4% of the LW population is mentally ill.
393 answer “yes” to at least one mental illness question. Lower bound: 26.1% of the LW population is mentally ill. Gosh, we have a lot of self-diagnosers.

Depression
Yes, I was formally diagnosed: 273, 18.2%
Yes, I self-diagnosed: 383, 25.5%
No: 759, 50.5%

OCD
Yes, I was formally diagnosed: 30, 2.0%
Yes, I self-diagnosed: 76, 5.1%
No: 1306, 86.9%

Autism spectrum

Yes, I was formally diagnosed: 98, 6.5%
Yes, I self-diagnosed: 168, 11.2%
No: 1143, 76.0%

Bipolar

Yes, I was formally diagnosed: 33, 2.2%
Yes, I self-diagnosed: 49, 3.3%
No: 1327, 88.3%

Anxiety disorder
Yes, I was formally diagnosed: 139, 9.2%
Yes, I self-diagnosed: 237, 15.8%
No: 1033, 68.7%

BPD
Yes, I was formally diagnosed: 5, 0.3%
Yes, I self-diagnosed: 19, 1.3%
No: 1389, 92.4%

[Ozy says: RATIONALIST BPDERS COME BE MY FRIEND]

Schizophrenia
Yes, I was formally diagnosed: 7, 0.5%
Yes, I self-diagnosed: 7, 0.5%
No: 1397, 92.9%

IV. Politics, Religion, Ethics

Politics
Communist: 9, 0.6%
Conservative: 67, 4.5%
Liberal: 416, 27.7%
Libertarian: 379, 25.2%
Social Democratic: 585, 38.9%

[The big change this year was that we changed "Socialist" to "Social Democratic". Even though the description stayed the same, about eight points worth of Liberals switched to Social Democrats, apparently more willing to accept that label than "Socialist". The overall supergroups Libertarian vs. (Liberal, Social Democratic) vs. Conservative remain mostly unchanged.]

Politics (longform)
Anarchist: 40, 2.7%
Communist: 9, 0.6%
Conservative: 23, 1.9%
Futarchist: 41, 2.7%
Left-Libertarian: 192, 12.8%
Libertarian: 164, 10.9%
Moderate: 56, 3.7%
Neoreactionary: 29, 1.9%
Social Democrat: 162, 10.8%
Socialist: 89, 5.9%

[Amusing politics answers include anti-incumbentist, having-well-founded-opinions-is-hard-but-I’ve-come-to-recognize-the-pragmatism-of-socialism-I-don’t-know-ask-me-again-next-year, pirate, progressive social democratic environmental liberal isolationist freedom-fries loving pinko commie piece of shit, republic-ist aka read the federalist papers, romantic reconstructionist, social liberal fiscal agnostic, technoutopian anarchosocialist (with moderate snark), whatever it is that Scott is, and WHY ISN’T THERE AN OPTION FOR NONE SO I CAN SIGNAL MY OBVIOUS OBJECTIVITY WITH MINIMAL EFFORT. Ozy would like to point out to the authors of manifestos that no one will actually read their manifestos except zir, and they might want to consider posting them to their own blogs.]


American Parties
Democratic Party: 221, 14.7%
Republican Party: 55, 3.7%
Libertarian Party: 26, 1.7%
Other party: 16, 1.1%
No party: 415, 27.6%
Non-Americans who really like clicking buttons: 415, 27.6%

Voting

Yes: 881, 58.6%
No: 444, 29.5%
My country doesn’t hold elections: 5, 0.3%

Religion

Atheist and not spiritual: 1054, 70.1%
Atheist and spiritual: 150, 10.0%
Agnostic: 156, 10.4%
Lukewarm theist: 44, 2.9%
Deist/pantheist/etc.: 22,, 1.5%
Committed theist: 60, 4.0%

Religious Denomination
Christian (Protestant): 53, 3.5%
Mixed/Other: 32, 2.1%
Jewish: 31, 2.0%
Buddhist: 30, 2.0%
Christian (Catholic): 24, 1.6%
Unitarian Universalist or similar: 23, 1.5%

[Amusing denominations include anti-Molochist, CelestAI, cosmic engineers, Laziness, Thelema, Resimulation Theology, and Pythagorean. The Cultus Deorum Romanorum practitioner still needs to contact Ozy so they can be friends.]

Family Religion
Atheist and not spiritual: 213, 14.2%
Atheist and spiritual: 74, 4.9%
Agnostic: 154. 10.2%
Lukewarm theist: 541, 36.0%
Deist/Pantheist/etc.: 28, 1.9%
Committed theist: 388, 25.8%

Religious Background
Christian (Protestant): 580, 38.6%
Christian (Catholic): 378, 25.1%
Jewish: 141, 9.4%
Christian (other non-protestant): 88, 5.9%
Mixed/Other: 68, 4.5%
Unitarian Universalism or similar: 29, 1.9%
Christian (Mormon): 28, 1.9%
Hindu: 23, 1.5%’

Moral Views
Accept/lean towards consequentialism: 901, 60.0%
Accept/lean towards deontology: 50, 3.3%
Accept/lean towards natural law: 48, 3.2%
Accept/lean towards virtue ethics: 150, 10.0%
Accept/lean towards contractualism: 79, 5.3%
Other/no answer: 239, 15.9%

Meta-ethics
Constructivism: 474, 31.5%
Error theory: 60, 4.0%
Non-cognitivism: 129, 8.6%
Subjectivism: 324, 21.6%
Substantive realism: 209, 13.9%

V. Community Participation


Less Wrong Use
Lurker: 528, 35.1%
I’ve registered an account: 221, 14.7%
I’ve posted a comment: 419, 27.9%
I’ve posted in Discussion: 207, 13.8%
I’ve posted in Main: 102, 6.8%

Sequences
Never knew they existed until this moment: 106, 7.1%
Knew they existed, but never looked at them: 42, 2.8%
Some, but less than 25%: 270, 18.0%
About 25%: 181, 12.0%
About 50%: 209, 13.9%
About 75%: 242, 16.1%
All or almost all: 427, 28.4%

Meetups
Yes, regularly: 154, 10.2%
Yes, once or a few times: 325, 21.6%
No: 989, 65.8%

Community

Yes, all the time: 112, 7.5%
Yes, sometimes: 191, 12.7%
No: 1163, 77.4%

Romance
Yes: 82, 5.5%
I didn’t meet them through the community but they’re part of the community now: 79, 5.3%
No: 1310, 87.2%

CFAR Events
Yes, in 2014: 45, 3.0%
Yes, in 2013: 60, 4.0%
Both: 42, 2.8%
No: 1321, 87.9%

CFAR Workshop
Yes: 109, 7.3%
No: 1311, 87.2%

[A couple percent more people answered 'yes' to each of meetups, physical interactions, CFAR attendance, and romance this time around, suggesting the community is very very gradually becoming more IRL. In particular, the number of people meeting romantic partners through the community increased by almost 50% over last year.]

HPMOR
Yes: 897, 59.7%
Started but not finished: 224, 14.9%
No: 254, 16.9%

Referrals
Referred by a link: 464, 30.9%
HPMOR: 385, 25.6%
Been here since the Overcoming Bias days: 210, 14.0%
Referred by a friend: 199, 13.2%
Referred by a search engine: 114, 7.6%
Referred by other fiction: 17, 1.1%

[Amusing responses include “a rationalist that I follow on Tumblr”, “I’m a student of tribal cultishness”, and “It is difficult to recall details from the Before Time. Things were brighter, simpler, as in childhood or a dream. There has been much growth, change since then. But also loss. I can't remember where I found the link, is what I'm saying.”]

Blog Referrals
Slate Star Codex: 40, 2.6%
Reddit: 25, 1.6%
Common Sense Atheism: 21, 1.3%
Hacker News: 20, 1.3%
Gwern: 13, 1.0%

VI. Other Categorical Data

Cryonics Status
Don’t understand/never thought about it: 62, 4.1%
Don’t want to: 361, 24.0%
Considering it: 551, 36.7%
Haven’t gotten around to it: 272, 18.1%
Unavailable in my area: 126, 8.4%
Yes: 64, 4.3%

Type of Global Catastrophic Risk
Asteroid strike: 64, 4.3%
Economic/political collapse: 151, 10.0%
Environmental collapse: 218, 14.5%
Nanotech/grey goo: 47, 3.1%
Nuclear war: 239, 15.8%
Pandemic (bioengineered): 310, 20.6%
Pandemic (natural): 113. 7.5%
Unfriendly AI: 244, 16.2%

[Amusing answers include ennui/eaten by Internet, Friendly AI, “Greens so weaken the rich countries that barbarians conquer us”, and Tumblr.]

Effective Altruism (do you self-identify)
Yes: 422, 28.1%
No: 758, 50.4%

[Despite some impressive outreach by the EA community, numbers are largely the same as last year]


Effective Altruism (do you participate in community)
Yes: 191, 12.7%
No: 987, 65.7%

Vegetarian
Vegan: 31, 2.1%
Vegetarian: 114, 7.6%
Other meat restriction: 252, 16.8%
Omnivore: 848, 56.4%

Paleo Diet

Yes: 33, 2.2%
Sometimes: 209, 13.9%
No: 1111, 73.9%

Food Substitutes
Most of my calories: 8. 0.5%
Sometimes: 101, 6.7%
Tried: 196, 13.0%
No: 1052, 70.0%

Gender Default
I only identify with my birth gender by default: 681, 45.3%
I strongly identify with my birth gender: 586, 39.0%

Books
<5: 198, 13.2%
5 - 10: 384, 25.5%
10 - 20: 328, 21.8%
20 - 50: 264, 17.6%
50 - 100: 105, 7.0%
> 100: 49, 3.3%

Birth Month
Jan: 109, 7.3%
Feb: 90, 6.0%
Mar: 123, 8.2%
Apr: 126, 8.4%
Jun: 107, 7.1%
Jul: 109, 7.3%
Aug: 120, 8.0%
Sep: 94, 6.3%
Oct: 111, 7.4%
Nov: 102, 6.8%
Dec: 106, 7.1%

[Despite my hope of something turning up here, these results don't deviate from chance]

Handedness
Right: 1170, 77.8%
Left: 143, 9.5%
Ambidextrous: 37, 2.5%
Unsure: 12, 0.8%

Previous Surveys
Yes: 757, 50.7%
No:  598, 39.8%

Favorite Less Wrong Posts (all > 5 listed)
An Alien God: 11
Joy In The Merely Real: 7
Dissolving Questions About Disease: 7
Politics Is The Mind Killer: 6
That Alien Message: 6
A Fable Of Science And Politics: 6
Belief In Belief: 5
Generalizing From One Example: 5
Schelling Fences On Slippery Slopes: 5
Tsuyoku Naritai: 5

VII. Numeric Data

Age: 27.67 + 8.679 (22, 26, 31) [1490]
IQ: 138.25 + 15.936 (130.25, 139, 146) [472]
SAT out of 1600: 1470.74 + 113.114 (1410, 1490, 1560) [395]
SAT out of 2400: 2210.75 + 188.94 (2140, 2250, 2320) [310]
ACT out of 36: 32.56 + 2.483 (31, 33, 35) [244]
Time in Community: 2010.97 + 2.174 (2010, 2011, 2013) [1317]
Time on LW: 15.73 + 95.75 (2, 5, 15) [1366]
Karma Score: 555.73 + 2181.791 (0, 0, 155) [1335]

P Many Worlds: 47.64 + 30.132 (20, 50, 75) [1261]
P Aliens: 71.52 + 34.364 (50, 90, 99) [1393]
P Aliens (Galaxy): 41.2 + 38.405 (2, 30, 80) [1379]
P Supernatural: 6.68 + 20.271 (0, 0, 1) [1386]
P God: 8.26 + 21.088 (0, 0.01, 3) [1376]
P Religion: 4.99 + 18.068 (0, 0, 0.5) [1384]
P Cryonics: 22.34 + 27.274 (2, 10, 30) [1399]
P Anti-Agathics: 24.63 + 29.569 (1, 10, 40) [1390]
P Simulation 24.31 + 28.2 (1, 10, 50) [1320]
P Warming 81.73 + 24.224 (80, 90, 98) [1394]
P Global Catastrophic Risk 72.14 + 25.620 (55, 80, 90) [1394]
Singularity: 2143.44 + 356.643 (2060, 2090, 2150) [1177]

[The mean for this question is almost entirely dependent on which stupid responses we choose to delete as outliers; the median practically never changes]


Abortion: 4.38 + 1.032 (4, 5, 5) [1341]
Immigration: 4 + 1.078 (3, 4, 5) [1310]
Taxes : 3.14 + 1.212 (2, 3, 4) [1410] (from 1 - should be lower to 5 - should be higher)
Minimum Wage: 3.21 + 1.359 (2, 3, 4) [1298] (from 1 - should be lower to 5 - should be higher)
Feminism: 3.67 + 1.221 (3, 4, 5) [1332]
Social Justice: 3.15 + 1.385 (2, 3, 4) [1309]
Human Biodiversity: 2.93 + 1.201 (2, 3, 4) [1321]
Basic Income: 3.94 + 1.087 (3, 4, 5) [1314]
Great Stagnation: 2.33 + .959 (2, 2, 3) [1302]
MIRI Mission: 3.90 + 1.062 (3, 4, 5) [1412]
MIRI Effectiveness: 3.23 + .897 (3, 3, 4) [1336]

[Remember, all of these are asking you to rate your belief in/agreement with the concept on a scale of 1 (bad) to 5 (great)]

Income: 54129.37 + 66818.904 (10,000, 30,800, 80,000) [923]
Charity: 1996.76 + 9492.71 (0, 100, 800) [1009]
MIRI/CFAR: 511.61 + 5516.608 (0, 0, 0) [1011]
XRisk: 62.50 + 575.260 (0, 0, 0) [980]
Older siblings: 0.51 + .914 (0, 0, 1) [1332]
Younger siblings: 1.08 + 1.127 (0, 1, 1) [1349]
Height: 178.06 + 11.767 (173, 179, 184) [1236]
Hours Online: 43.44 + 25.452 (25, 40, 60) [1221]
Bem Sex Role Masculinity: 42.54 + 9.670 (36, 42, 49) [1032]
Bem Sex Role Femininity: 42.68 + 9.754 (36, 43, 50) [1031]
Right Hand: .97 + 0.67 (.94, .97, 1.00)
Left Hand: .97 + .048 (.94, .97, 1.00)

VIII. Fishing Expeditions

[correlations, in descending order]

SAT Scores out of 1600/SAT Scores out of 2400 .844 (59)
P Supernatural/P God .697 (1365)
Feminism/Social Justice .671 (1299)
P God/P Religion .669 (1367)
P Supernatural/P Religion .631 (1372)
Charity Donations/MIRI and CFAR Donations .619 (985)
P Aliens/P Aliens 2 .607 (1376)
Taxes/Minimum Wage .587 (1287)
SAT Score out of 2400/ACT Score .575 (89)
Age/Number of Children .506 (1480)
P Cryonics/P Anti-Agathics .484 (1385)
SAT Score out of 1600/ACT Score .480 (81)
Minimum Wage/Social Justice .456 (1267)
Taxes/Social Justice .427 (1281)
Taxes/Feminism .414 (1299)
MIRI Mission/MIRI Effectiveness .395 (1331)
P Warming/Taxes .385 (1261)
Taxes/Basic Income .383 (1285)
Minimum Wage/Feminism .378 (1286)
P God/Abortion -.378 (1266)
Immigration/Feminism .365 (1296)
P Supernatural/Abortion -.362 (1276)
Feminism/Human Biodiversity -.360 (1306)
MIRI and CFAR Donations/Other XRisk Charity Donations .345 (973)
Social Justice/Human Biodiversity -.341 (1288)
P Religion/Abortion -.326 (1275)
P Warming/Minimum Wage .324 (1248)
Minimum Wage/Basic Income .312 (1276)
P Warming/Basic Income .306 (1260)
Immigration/Social Justice .294 (1278)
P Anti-Agathics/MIRI Mission .293 (1351)
P Warming/Feminism .285 (1281)
P Many Worlds/P Anti-Agathics .276 (1245)
Social Justice/Femininity .267 (990)
Minimum Wage/Human Biodiversity -.264 (1274)
Immigration/Human Biodiversity -.263 (1286)
P Many Worlds/MIRI Mission .263 (1233)
P Aliens/P Warming .262 (1365)
P Warming/Social Justice .257 (1262)
Taxes/Human Biodiversity -.252 (1291)
Social Justice/Basic Income .251 (1281)
Feminism/Femininity .250 (1003)
Older Siblings/Younger Siblings -.243 (1321)
Charity Donations/Other XRisk Charity Donations .240 (957
P Anti-Agathics/P Simulation .238 (1312)
Abortion/Minimum Wage .229 (1293)
Feminism/Basic Income .227 (1297)
Abortion/Feminism .226 (1321)
P Cryonics/MIRI Mission .223 (1360)
Immigration/Basic Income .208 (1279)
P Many Worlds/P Cryonics .202 (1251)
Number of Current Partners/Femininity: .202 (1029)
P Warming/Immigration .202 (1260)
P Warming/Abortion .201 (1289)
Abortion/Taxes .198 (1304)
Age/P Simulation .197 (1313)
Political Interest/Masculinity .194 (1011)
P Cryonics/MIRI Effectiveness .191 (1285)
Abortion/Social Justice .191 (1301)
P Simulation/MIRI Mission .188 (1290)
P Many Worlds/P Warming .188 (1240)
Age/Number of Current Partners .184 (1480)
P Anti-Agathics/MIRI Effectiveness .183 (1277)
P Many Worlds/P Simulation .181 (1211)
Abortion/Immigration .181 (1304)
Number of Current Partners/Number of Children .180 (1484)
P Cryonics/P Simulation .174 (1315)
P Global Catastrophic Risk/MIRI Mission -.174 (1359)
Minimum Wage/Femininity .171 (981)
Abortion/Basic Income .170 (1302)
Age/P Cryonics -.165 (1391)
Immigration/Taxes .165 (1293)
P Warming/Human Biodiversity -.163 (1271)
P Aliens 2/Warming .160 (1353)
Abortion/Younger Siblings -.155 (1292)
P Religion/Meditate .155 (1189)
Feminism/Masculinity -.155 (1004)
Immigration/Femininity .155 (988)
P Supernatural/Basic Income -.153 (1246)
P Supernatural/P Warming -.152 (1361)
Number of Current Partners/Karma Score .152 (1332)
P Many Worlds/MIRI Effectiveness .152 (1181)
Age/MIRI Mission -.150 (1404)
P Religion/P Warming -.150 (1358)
P Religion/Basic Income -.146 (1245)
P God/Basic Income -.146 (1237)
Human Biodiversity/Femininity -.145 (999)
P God/P Warming -.144 (1351)
Taxes/Femininity .142 (987)
Number of Children/Younger Siblings .138 (1343)
Number of Current Partners/Masculinity: .137 (1030)
P Many Worlds/P God -.137 (1232)
Age/Charity Donations .133 (1002)
P Anti-Agathics/P Global Catastrophic Risk -.132 (1373)
P Warming/Masculinity -.132 (992)
P Global Catastrophic Risk/MIRI and CFAR Donations -.132 (982)
P Supernatural/Singularity .131 (1148)
God/Taxes -.130 (1240)
Age/P Anti-Agathics -.128 (1382)
P Aliens/Taxes .127(1258)
Feminism/Great Stagnation -.127 (1287)
P Many Worlds/P Supernatural -.127 (1241)
P Aliens/Abortion .126 (1284)
P Anti-Agathics/Great Stagnation -.126 (1248)
P Anti-Agathics/P Warming .125 (1370)
Age/P Aliens .124 (1386)
P Aliens/Minimum Wage .124 (1245)
P Aliens/P Global Catastrophic Risk .122 (1363)
Age/MIRI Effectiveness -.122 (1328)
Age/P Supernatural .120 (1370)
P Supernatural/MIRI Mission -.119 (1345)
P Many Worlds/P Religion -.119 (1238)
P Religion/MIRI Mission -.118 (1344)
Political Interest/Social Justice .118 (1304)
P Anti-Agathics/MIRI and CFAR Donations .118 (976)
Human Biodiversity/Basic Income -.115 (1262)
P Many Worlds/Abortion .115 (1166)
Age/Karma Score .114 (1327)
P Aliens/Feminism .114 (1277)
P Many Worlds/P Global Catastrophic Risk -.114 (1243)
Political Interest/Femininity .113 (1010)
Number of Children/P Simulation -.112 (1317)
P Religion/Younger Siblings .112 (1275)
P Supernatural/Taxes -.112 (1248)
Age/Masculinity .112 (1027)
Political Interest/Taxes .111 (1305)
P God/P Simulation .110 (1296)
P Many Worlds/Basic Income .110 (1139)
P Supernatural/Younger Siblings .109 (1274)
P Simulation/Basic Income .109 (1195)
Age/P Aliens 2 .107 (1371)
MIRI Mission/Basic Income .107 (1279)
Age/Great Stagnation .107 (1295)
P Many Worlds/P Aliens .107 (1253)
Number of Current Partners/Social Justice .106 (1304)
Human Biodiversity/Great Stagnation .105 (1285)
Number of Children/Abortion -.104 (1337)
Number of Current Partners/P Cryonics -.102 (1396)
MIRI Mission/Abortion .102 (1305)
Immigration/Great Stagnation -.101 (1269)
Age/Political Interest .100 (1339)
P Global Catastrophic Risk/Political Interest .099 (1295)
P Aliens/P Religion -.099 (1357)
P God/MIRI Mission -.098 (1335)
P Aliens/P Simulation .098 (1308)
Number of Current Partners/Immigration .098 (1305)
P God/Political Interest .098 (1274)
P Warming/P Global Catastrophic Risk .096 (1377)

In addition to the Left/Right factor we had last year, this data seems to me to have an Agrees with the Sequences Factor-- the same people tend to believe in many-worlds, cryo, atheism, simulationism, MIRI’s mission and effectiveness, anti-agathics, etc. Weirdly, belief in global catastrophic risk is negatively correlated with most of the Agrees with Sequences things. Someone who actually knows how to do statistics should run a factor analysis on this data.

IX. Digit Ratios

After sanitizing the digit ratio numbers, the following correlations came up:

Digit ratio R hand was correlated with masculinity at a level of -0.180 p < 0.01
Digit ratio L hand was correlated with masculinity at a level of -0.181 p < 0.01
Digit ratio R hand was slightly correlated with femininity at a level of +0.116 p < 0.05

Holy #@!$ the feminism thing ACTUALLY HELD UP. There is a 0.144 correlation between right-handed digit ratio and feminism, p < 0.01. And an 0.112 correlation between left-handed digit ratio and feminism, p < 0.05.

The only other political position that correlates with digit ratio is immigration. There is a 0.138 correlation between left-handed digit ratio and believe in open borders p < 0.01, and an 0.111 correlation between right-handed digit ratio and belief in open borders, p < 0.05.

No digit correlation with abortion, taxes, minimum wage, social justice, human biodiversity, basic income, or great stagnation.

Okay, need to rule out that this is all confounded by gender. I ran a few analyses on men and women separately.

On men alone, the connection to masculinity holds up. Restricting sample size to men, left-handed digit ratio corresponds to masculinity with at -0.157, p < 0.01. Left handed at -0.134, p < 0.05. Right-handed correlates with femininity at 0.120, p < 0.05. The feminism correlation holds up. Restricting sample size to men, right-handed digit ratio correlates with feminism at a level of 0.149, p < 0.01. Left handed just barely fails to correlate. Both right and left correlate with immigration at 0.135, p < 0.05.

On women alone, the Bem masculinity correlation is the highest correlation we're going to get in this entire study. Right hand is -0.433, p < 0.01. Left hand is -0.299, p < 0.05. Femininity trends toward significance but doesn't get there. The feminism correlation trends toward significance but doesn't get there. In general there was too small a sample size of women to pick up anything but the most whopping effects.

Since digit ratio is related to testosterone and testosterone sometimes affects risk-taking, I wondered if it would correlate with any of the calibration answers. I selected people who had answered Calibration Question 5 incorrectly and ran an analysis to see if digit ratio was correlated with tendency to be more confident in the incorrect answer. No effect was found.

Other things that didn't correlate with digit ratio: IQ, SAT, number of current partners, tendency to work in mathematical professions.

...I still can't believe this actually worked. The finger-length/feminism connection ACTUALLY WORKED. What a world. What a world. Someone may want to double-check these results before I get too excited.

X. Calibration


There were ten calibration questions on this year's survey. Along with answers, they were:

1. What is the largest bone in the body? Femur
2. What state was President Obama born in? Hawaii
3. Off the coast of what country was the battle of Trafalgar fought? Spain
4. What Norse God was called the All-Father? Odin
5. Who won the 1936 Nobel Prize for his work in quantum physics? Heisenberg
6. Which planet has the highest density? Earth
7. Which Bible character was married to Rachel and Leah? Jacob
8. What organelle is called "the powerhouse of the cell"? Mitochondria
9. What country has the fourth-highest population? Indonesia
10. What is the best-selling computer game? Minecraft

I ran calibration scores for everybody based on how well they did on the ten calibration questions. These failed to correlate with IQ, SAT, LW karma, or any of the things you might expect to be measures of either intelligence or previous training in calibration; they didn't differ by gender, correlates of community membership, or any mental illness [deleted section about correlating with MWI and MIRI, this was an artifact].

Your answers looked like this:



The red line represents perfect calibration. Where answers dip below the line, it means you were overconfident; when they go above, it means you were underconfident.

It looks to me like everyone was horrendously underconfident on all the easy questions, and horrendously overconfident on all the hard questions. To give an example of how horrendous, people who were 50% sure of their answers to question 10 got it right only 13% of the time; people who were 100% sure only got it right 44% of the time. Obviously those numbers should be 50% and 100% respectively.

This builds upon results from previous surveys in which your calibration was also horrible. This is not a human universal - people who put even a small amount of training into calibration can become very well calibrated very quickly. This is a sign that most Less Wrongers continue to neglect the very basics of rationality and are incapable of judging how much evidence they have on a given issue. Veterans of the site do no better than newbies on this measure.

XI. Wrapping Up

To show my appreciation for everyone completing this survey, including the arduous digit ratio measurements, I have randomly chosen a person to receive a $30 monetary prize. That person is...the person using the public key "The World Is Quiet Here". If that person tells me their private key, I will give them $30.

I have removed 73 people who wished to remain private, deleted the Private Keys, and sanitized a very small amount of data. Aside from that, here are the raw survey results for your viewing and analyzing pleasure:

(as Excel)

(as SPSS)

(as CSV)

Comment author: Yvain 20 December 2014 04:41:12AM 12 points [-]

Smell is interesting because it's way overrepresented genetically. Something like 5% of most animals' genomes are just a whole bunch of olfactory receptor genes, each for a different individual smell. So it should be unusually easy to do epigenetics with it. Just say "Express the gene for cherry smell more" and then the mice have a stronger reaction to it.

This doesn't mean that any more complex behaviors can be inherited epigenetically. In fact, it might be that nothing else is as suitable to epigenetic transmission as olfaction.

Comment author: Yvain 08 December 2014 01:26:56AM 75 points [-]

Both accounts' SSC comments come from the same IP.

Comment author: Brillyant 21 November 2014 07:14:07PM 6 points [-]

In his defense, is it possible EY can't win at this point, regardless of his approach? Maybe the internet has grabbed this thing and the PR whirlwinds are going to do with it whatever they like?

I've read apologies from EY where he seems to admit pretty clearly he screwed up. He comes off as defensive and pissy sometimes in my opinion, but he seems sincerely irked about how RW and other outlets have twisted to whole story to discredit LW and himself. From my recall, one comment he made on the reddit sub dedicated to his HP fanfic indicated he was very hurt by the whole kerfuffle, in addition to his obvious frustration.

Comment author: Yvain 22 November 2014 02:39:54AM *  51 points [-]

At this point I think the winning move is rolling with it and selling little plush basilisks as a MIRI fundraiser. It's our involuntary mascot, and we might as well 'reclaim' it in the social justice sense.

Then every time someone brings up "Less Wrong is terrified of the basilisk" we can just be like "Yes! Yes we are! Would you like to buy a plush one?" and everyone will appreciate our ability to laugh at ourselves, and they'll go back to whatever they were doing.

Comment author: Lumifer 22 November 2014 02:23:02AM 5 points [-]

Does MIRI actually has a basement?

It's behind the hidden door. Full of boxes which say "AI inside -- DO NOT TALK TO IT".

The ghosts there are not really dangerous. Usually.

Comment author: Yvain 22 November 2014 02:37:11AM *  26 points [-]

When I visited MIRI's headquarters, they were trying to set up a video link to the Future of Humanity Institute. Somebody had put up a monitor in a prominent place and there was a sticky note saying something like "Connects to FHI - do not touch".

Except that the H was kind of sloppy and bent upward so it looked like an A.

I was really careful not to touch that monitor.

Comment author: Toggle 18 November 2014 04:42:06AM 14 points [-]

It's curious to see the frequency of posts that start with "I am not a neoreactionary, but...". (This includes my own). If I'm not mistaken, they seem to outnumber the actual neoreactionary posts by a fair margin.

I think a call for patriarchal racially-stratified monarchy is catnip around here. Independently of its native virtues, I mean. It's a debate that couldn't even happen in most communities, so it's reinforcing our sense of LW's peculiar set of community mores. It's a radical but also unexpected vision of a technological future, so it has new ideas to wrestle with, and enough in the way of historical roots to reward study and give all participants the chance to learn. And it is political without being ossified in to tired and nationally televised debates, with new insights available to a clever thinker and plenty of room to pull sideways.

For that reason, I'm a little worried that it will receive disproportionate attention. I know my System 1 loves to read the stuff. But System 2... Enthusiastic engagement with political monarchy- pro or con- is not something I would like to see become a major feature of Less Wrong, so I think I'm going to publicly commit to posting no more than one NRx comment per month, pending major changes in community dynamics.

Comment author: Yvain 21 November 2014 08:02:34AM 20 points [-]

I agree with Toggle that this might not have been the best place for this question.

The Circle of Life goes like this. Somebody associates Less Wrong with neoreactionaries, even though there are like ten of them here total. They start discussing neoreaction here, or asking their questions for neoreactionaries here. The discussion is high profile and leads more people to associate Less Wrong with neoreactionaries. That causes more people to discuss it and ask questions here, which causes more people to associate us, and it ends with everybody certain that we're full of neoreactionaries, and that ends with bad people who want to hurt us putting "LESS WRONG IS A RACIST NEOREACTIONARY WEBSITE" in big bold letters over everything.

If you really want to discuss neoreaction, I'd suggest you do it in an Slate Star Codex open thread, since apparently I'm way too tarnished by association with them to ever escape. Or you can go to a Xenosystems open thread and get it straight from the horse's mouth.

Comment author: Lumifer 29 October 2014 08:07:00PM *  3 points [-]

Nobody actually buys Minesweeper, so I don't think it counts as a bestselling game.

Having said this, the claims about the bestselling game of all time upthread sound wrong to me. The first game that came to mind, Wikipedia says this about it:

Va Wnahnel 2010, vg jnf naabhaprq gung gur Grgevf senapuvfr unq fbyq zber guna 170 zvyyvba pbcvrf, nccebkvzngryl 70 zvyyvba culfvpny pbcvrf naq bire 100 zvyyvba pbcvrf sbe pryy cubarf,[9][10] znxvat vg gur uvturfg cnvq-qbjaybnqrq tnzr bs nyy gvzr.

which handily beats Minecraft.

Comment author: Yvain 02 November 2014 01:52:17AM 2 points [-]

I stated that all disputes would be resolved by Wikipedia, and here is Wikipedia's verdict on the matter: http://en.wikipedia.org/wiki/List_of_best-selling_PC_games

Meetup : Michigan Meetup 11/9

1 Yvain 31 October 2014 12:42AM

Discussion article for the meetup : Michigan Meetup 11/9

WHEN: 09 November 2014 03:00:00PM (-0400)

WHERE: 19334 Angling Street, Livonia, MI

Our more-or-less bimonthly Ann Arbor + Detroit area meetup. Myself and Ozy will be hosting.

No particular topic, but bring games and discussion topics as they interest you.

Discussion article for the meetup : Michigan Meetup 11/9

Comment author: queeborg 24 October 2014 12:07:42PM 42 points [-]

Survey completed. Account created to get starting karma and increase likelihood/amount of future participation.

I'd like to note that the current formulation of sex/gender/sexual orientation questions forced me to misrepresent myself because the technically correct answers seemed to cause an even greater misrepresentation. I would like extra options to the "sex assigned at birth" question, perhaps "male, now transitioned to female/other" and vice versa, to account for other-gendered transitioners; but I'll be the first to admit that this probably isn't a major issue.

Comment author: Yvain 26 October 2014 08:07:59PM 9 points [-]

I'm confused. If you were male at birth and transitioned to female, can't you just answer the "sex assigned at birth" question male, and the gender question with "transgender m -> f" ?

Comment author: SteveReilly 26 October 2014 06:41:20PM *  37 points [-]

Finished it. I can't wait to read the post that talks about how bad people are at following directions.

Comment author: Yvain 26 October 2014 08:06:02PM 9 points [-]

I can already tell you that...well, you remember the preview thread. The one where I posted a version of the survey saying in big letters on the top "DO NOT TAKE THIS, IT IS NOT OPEN" and the first question was "You are not supposed to take the survey now" and the only answer was "Okay, I'll stop"?

Four people took it. Obviously they won't be counted.

Comment author: Vulture 24 October 2014 06:31:26PM 33 points [-]

Something that just occurred to me (separate from my took-it comment): Scott, do you take your own survey?

Comment author: Yvain 26 October 2014 08:03:58PM 7 points [-]

Yes, but I keep my data private because I'd be easy to find otherwise and I don't want everyone knowing my income and politics et cetera.

2014 Less Wrong Census/Survey

88 Yvain 26 October 2014 06:05PM

It's that time of year again.

If you are reading this post and self-identify as a LWer, then you are the target population for the Less Wrong Census/Survey. Please take it. Doesn't matter if you don't post much. Doesn't matter if you're a lurker. Take the survey.

This year's census contains a "main survey" that should take about ten or fifteen minutes, as well as a bunch of "extra credit questions". You may do the extra credit questions if you want. You may skip all the extra credit questions if you want. They're pretty long and not all of them are very interesting. But it is very important that you not put off doing the survey or not do the survey at all because you're intimidated by the extra credit questions.

It also contains a chance at winning a MONETARY REWARD at the bottom. You do not need to fill in all the extra credit questions to get the MONETARY REWARD, just make an honest stab at as much of the survey as you can.

Please make things easier for my computer and by extension me by reading all the instructions and by answering any text questions in the simplest and most obvious possible way. For example, if it asks you "What language do you speak?" please answer "English" instead of "I speak English" or "It's English" or "English since I live in Canada" or "English (US)" or anything else. This will help me sort responses quickly and easily. Likewise, if a question asks for a number, please answer with a number such as "4", rather than "four".

The planned closing date for the survey is Friday, November 14. Instead of putting the survey off and then forgetting to do it, why not fill it out right now?

Okay! Enough preliminaries! Time to take the...

***


[EDIT: SURVEY CLOSED, DO NOT TAKE!]

***

Thanks to everyone who suggested questions and ideas for the 2014 Less Wrong Census/Survey. I regret I was unable to take all of your suggestions into account, because of some limitations in Google Docs, concern about survey length, and contradictions/duplications among suggestions. The current survey is a mess and requires serious shortening and possibly a hard and fast rule that it will never get longer than it is right now.

By ancient tradition, if you take the survey you may comment saying you have done so here, and people will upvote you and you will get karma.

Comment author: Yvain 15 October 2014 07:30:05AM *  18 points [-]

Should effective altruists donate to fighting Ebola?

Argument against: usually very famous things that make the news are terrible effective altruist causes and you should stick to well-studied things like malaria.

Argument for: Ebola is very underfunded compared to sexier disasters. And it is a disease in the Third World, a category which has brought us most of the best-known effective altruism interventions.

Thoughts: The CDC estimates a best-case scenario of 20,000 cases by January and a worst-case scenario of about 1.5 million cases by January. They do not estimate risks past January. There are also black swan risks in which Ebola spreads to the entire Third World (eg India) and kills tens of millions of people there. However, on the margin individual donations are unlikely to shift the virus from one of these scenarios to another, so it's probably more worth considering how much good the marginal donation does.

Doctors Without Borders is a very well known, GiveWell-approved charity. They are running clinics in the country, but it's hard to tell how much more clinic they can run per dollar. On the other hand, they are also giving out home infection prevention kits by the tens of thousands. Other charities price these at about ten dollars per kit, although I've seen estimates that differ by an order of magnitude. I don't think anybody knows how effective the kits are going to be, although everyone agrees they are a vastly inferior option to sufficient space in hospitals, which at the moment does not exist.

If we estimate likelihood of 100,000 Liberians (geometric mean of estimates) eventually infected = 2% of the population, then $1000 buys 100 kits buys 2 kits for people likely to be infected..

$1000 for malaria bed nets supposedly gives something like 20 to 100 DALYs, depending on whose estimate you trust.

Ebola death rate is about 50%. Suppose the average infected person has 30 DALYs left to lose. So each case of Ebola costs 15 DALYs directly. But it probably ends up costing more like 30, because I think on average each case infects one other person (I don't think this is meant to be iterate, or else the estimate quickly goes to infinity). So if every Ebola kit was 100% effective, we would expect distributing the kits to save 60 DALYs.

That means in order for kits to be as good as the bottom range of estimates for bed nets, they would have to be at least 33% effective in preventing Ebola among people who get them, which they probably aren't.

On the other hand, every number in this estimate is a total wild guess, and I don't trust that I'm within two orders of magnitude of anything approaching reality. Kits likely cost more when including distribution (I expect charities to underreport costs to make people feel good about giving them), there's no guarantee that there's room for more kits, and my rate of how many subsequent cases are caused by each case is from a half-remembered news article. Does anyone have better ideas for how to figure this out?

Comment author: DanielFilan 11 October 2014 01:43:58PM *  4 points [-]

Formatting issues:

  • The title "Part Eight: Slightly More Complicated Questions" appears twice.

Question requests:

  • Ability to solve the Schrodinger equation for the hydrogen atom.
  • OCEAN personality test results
  • Split "no" option in meetups into "no, because there are no meetups near where I live" and "no, there may be meetups near where I live but I don't want to go to them"

Other comments:

  • I like the multiple calibration questions
Comment author: Yvain 12 October 2014 03:37:24AM 3 points [-]

There was a Schrodinger atom question a couple years ago. I'm trying not to keep all questions lest the survey just grow and grow forever. Any particular reason you want to know whether the Schrodinger solving percent has changed since last time?

Comment author: ShardPhoenix 11 October 2014 09:06:32AM *  8 points [-]

I feel like using Scandanavia as an example of "socialism" is not really accurate - they're capitalist welfare states with slightly higher taxes than other capitalist welfare states.

Comment author: Yvain 12 October 2014 03:36:09AM 3 points [-]

How would you handle this?

Comment author: [deleted] 11 October 2014 11:31:45AM 5 points [-]
  • Given that you already ask the Sex question, the Gender question can have just three answers “Man”, “Woman”, and “Neither/Other”.
  • As suggested by people on SSC, you might want to add a question “Is your gender an important part of your identity?” or something, to find out how many of us are cis-by-default.
  • I'd have three answers to Relationship Goals, “actively looking”, “open but not actively looking”, and “not open”.
  • Split the “no” answer to Children into “not now” and “never”.
  • Selfishly, I'm glad you didn't split physics in the Profession question into theoretical physics and experimental physics so that I, as a phenomenologist, don't have to choose, but you might still want to consider splitting them.
  • Maybe in the Religious Views question “Agnostic” → “Agnostic/Apatheist/Ignostic” (cf. “Don't know/Don't care/Don't understand”). Also, consider specifying what the hell you mean by “spiritual”, as IIRC the previous times different people interpreted it in different ways.
  • In the Aliens question, does octopus-level intelligence count? Does pre- human-level intelligence count?
  • Wouldn't (say) three calibration questions be enough?
Comment author: Yvain 12 October 2014 03:35:52AM 9 points [-]

Anything I do with gender and sex is going to have lots of people yell at me. But if I keep it the same, it will be the same people as last year and I won't make new enemies.

Comment author: Princess_Stargirl 11 October 2014 06:06:15PM *  9 points [-]

I would personally be interested in more detailed drug use questions. Examplae. In the Past year have you taken:

-Modafinil

-Amphetamine (of any sort including Adderall)

-Heroine/Other Opiates (not prescribed by a doctor as a painkiller) -Marijuanna

-Hallucinogens (LSD/DMT/Psychobillin)

-Testosterone/HGH (if you took testosterone because you are transexual do not click yes)

Maybe there is a better list of drugs?

Comment author: Yvain 12 October 2014 03:35:01AM 7 points [-]

I had this last time, and several people told me to take it off because it was bad to make people admit to illegal activities.

Also, for complicated reasons I can't do "Check as many as apply" questions, so this would take forever.

Comment author: gwern 11 October 2014 05:41:29PM *  29 points [-]
  • I suggest a basilisk question, as usual.
  • I'd also like OCEAN/Big Five restored. The results from last time weren't very definitive and more data would be helpful.
  • Given the recent SSC debate on how effective MIRI is, it might be interesting to ask something about opinions of MIRI's effectiveness in pursuing its mission: just asking how much people donated is insufficient since one might donate faute de mieux, and IIRC the modal donation is $0 in the first place.
  • Perhaps the AI projection questions could be replaced by the same question-set used in "Future progress in artificial intelligence: A poll among experts", VC Müller, N Bostrom? Buys one comparability with their survey responses; it would be very interesting and intriguing if LWers turned out to be less extreme.
Comment author: Yvain 12 October 2014 03:34:04AM 4 points [-]

Any particular implementation details on OCEAN? Exact same as last time?

Comment author: ChristianKl 11 October 2014 04:15:14PM 1 point [-]

I oppose the Birth month question. It costs a lot of anonymity but provides no good return.

Comment author: Yvain 11 October 2014 05:24:57PM 5 points [-]

2014 Less Wrong Census/Survey - Call For Critiques/Questions

18 Yvain 11 October 2014 06:39AM

It's that time of year again. Actually, a little earlier than that time of year, but I'm pushing it ahead a little to match when Ozy and I expect to have more free time to process the results.

The first draft of the 2014 Less Wrong Census/Survey is complete (see 2013 results here) .

You can see the survey below if you promise not to try to take the survey because it's not done yet and this is just an example!

2014 Less Wrong Census/Survey Draft

I want two things from you.

First, please critique this draft (it's much the same as last year's). Tell me if any questions are unclear, misleading, offensive, confusing, or stupid. Tell me if the survey is so unbearably long that you would never possibly take it. Tell me if anything needs to be rephrased.

Second, I am willing to include any question you want in the Super Extra Bonus Questions section, as long as it is not offensive, super-long-and-involved, or really dumb. Please post any questions you want there. Please be specific - not "Ask something about taxes" but give the exact question you want me to ask as well as all answer choices.

Try not to add more than a few questions per person, unless you're sure yours are really interesting. Please also don't add any questions that aren't very easily sort-able by a computer program like SPSS unless you can commit to sorting the answers yourself.

I will probably post the survey to Main and officially open it for responses sometime early next week.

In response to Questions on Theism
Comment author: gjm 08 October 2014 10:24:09PM 28 points [-]

So, a few observations on miracles.

  1. There are miracle stories in every religious tradition and plenty of not-exactly-religious traditions. Unless there's some big difference in credibility -- which I'm not aware of any reason to think there is -- if you think "no smoke without fire" about one set then you should think the same about the others too. Which means you either have to believe in lots of different gods, or believe in one god and lots of evil spirits (or something) that just happen to do more or less the same sorts of miracle. (Or, I guess, believe that miraculous things happen but they're brought about by people's latent psychic powers or something, but that's pretty far from any religion's account of these things.)

  2. When miraculous stories are investigated carefully, they consistently seem to evaporate. This happens even when the people doing the investigation belong to the religion that claims responsibility for the alleged miracle. For instance, consider something commonly cited as evidence for miracles: the shrine at Lourdes, to which pilgrims in their millions trek in the hope of miraculous healing. The Roman Catholic Church has a process -- to its credit, not a completely ridiculous one -- by which it certifies some healings there as miraculous. Although the process isn't completely ridiculous, it's far from obviously bulletproof; the main requirement is that a bunch of Roman Catholic doctors declare that the alleged cure is inexplicable according to current medical knowledge. As an example, the most recent case is of someone who had a tumour that went away after she bathed at Lourdes. (My understanding is that this is a thing that occasionally happens, miracle or no.) So, anyway, they appear to certify about one miracle per two million pilgrims, and I think pretty much all the pilgrims are there in hope of healing. One per two million! (If you think the alleged cures are so improbable that they couldn't happen naturally one time in two million, I have a bridge to sell you.)

  3. In some situations (those in which a lot of these miraculous healings tend to occur) it really isn't difficult to get people to think more has happened than really has. Consider, for instance, the case of Peter Popoff. Lots of miraculous healings at his meetings -- but the whole thing was a fraud.

  4. In general, unfortunately, people do lie. And make mistakes. And see what they hope or expect to see. And tales "grow in the telling", so that after a few steps of Chinese Whispers something sounds far more inexplicable and impressive than it ever really was.

You might try the following experiment: Talk to some of your Christian friends, and ask them for the most impressive examples they have personally experienced of miraculous interventions by God. If in fact there are no miracles, what you should expect is that (1) the things they cite won't, on the whole, be all that impressive; (2) the more careful and intelligent of them will have less impressive experiences; (3) the most impressive experiences will be the least verifiable.

In response to comment by gjm on Questions on Theism
Comment author: Yvain 09 October 2014 04:25:06AM *  41 points [-]

The Roman Catholic Church has a process -- to its credit, not a completely ridiculous one -- by which it certifies some healings there as miraculous. Although the process isn't completely ridiculous, it's far from obviously bulletproof; the main requirement is that a bunch of Roman Catholic doctors declare that the alleged cure is inexplicable according to current medical knowledge.

I went to medical school in Ireland and briefly rotated under a neurologist there. One time he received a very nice letter from the Catholic Church, saying that one of his patients had gotten much better after praying to a certain holy figure, and the Church was trying to canonize (or beatify, or whatever) the figure, so if the doctor could just certify that the patient's recovery was medically impossible, that would be really helpful and make everyone very happy.

The neurologist wrote back that the patient had multiple sclerosis, a disease which remits for long periods on its own all the time and so there was nothing medically impossible about the incident at all.

I have only vague memories of this, but I think the Church kept pushing it, asking whether maybe it was at least a little medically impossible, because they really wanted to saint this guy.

(the neurologist was an atheist and gleefully refused as colorfully as he could)

This left me less confident in accounts of medical miracles.

Comment author: Yvain 17 September 2014 12:35:14AM 7 points [-]

Hello. I run a Michigan meetup group based in Livonia. We have about fifteen people from the Detroit and Ann Arbor areas. If you give me your email, I will add you to our list.

I don't remember anyone who I'm sure is at the university in particular right now except for Sniffnoy. You can private message him here under that name, or I can relay a message if you want.

Comment author: Unnamed 22 July 2014 06:31:57AM 13 points [-]

How about "politics is a minefield"?

I see that shminux & Yvain have already used that phrase in their comments.

Comment author: Yvain 24 July 2014 01:08:57AM 0 points [-]

That sounds like a pretty good compromise.

Comment author: RobbBB 23 July 2014 11:03:36AM *  1 point [-]

It sounds like we agree it'd be bad for LW to go political, but we're worrying about different scenarios. Some of my concerns:

  • 'Politics is the mind-killer', as most people use it, carries approximately the same content as 'boo politics'. If one of LW's top catchphrases is 'boo politics!', we're more likely to alienate people with the connections and expertise needed to handle politically charged blow-ups, group dynamics, etc. well. From what I can tell, when organizations, communities, and movements avoid getting dragged through the mud due to misinformation being circulated online, it's frequently because they have friends who are skilled or connected e.g. at social media, diplomacy / PR.

Having fully general counterarguments against your hated enemies, and lots of blog posts readying your troops for battle with those hated enemies, is not generally a winning way to avoid getting into lots of messy time-wasting fights. On my understanding, cultivating targeted social skills/habits (for preventing, diffusing, and redirecting conflict) and allies/connections works better.

  • If one of LW's top catchphrases is 'boo politics!', we'll thereby by setting ourselves up as the Anti-Politics Tribe, a hated enemy of the Politics Tribes. The Politics Tribes are precisely the people we're trying to avoid picking fights with, especially not fights framed as tribalistic no-holds-barred absolutist sloganeering shouting matches.

Going meta is not a secure safeguard; it just means that any political partisan or activist community can potentially object or take offense, since we're now talking about politics as a totality.

most of the people who say we're "arrogant" will find a reason to think so no matter how we phrase things.

That's not my experience, but if that's true, then a lot of the people I'm interested in building ties to are in that high-value has-a-nuanced-position minority. My own opinion of LW shifts up and down by increments based on how nice I see people being, and I see a lot of my friends fluctuating up and down in opinion based on incidents like 'this person condescended to me', 'I read this extremely insightful blog post', etc.

Comment author: Yvain 24 July 2014 01:05:22AM *  12 points [-]

Politics is the mind-killer', as most people use it, carries approximately the same content as 'boo politics'. If one of LW's top catchphrases is 'boo politics!', we're more likely to alienate people with the connections and expertise needed to handle politically charged blow-ups, group dynamics, etc. well.

I think this conflates "people who are good at group dynamics" and "people who argue a lot about abortion" into the category "politics people". I doubt there is much of a correlation between the two categories. If we really wanted people who were good at handling these sorts of things, I would look for business managers, sports team captains, and people with nonprofit experience before I started looking for people marked by an interest in politics.

From what I can tell, when organizations, communities, and movements avoid getting dragged through the mud due to misinformation being circulated online, it's frequently because they have friends who are skilled or connected e.g. at social media, diplomacy / PR.

Huh. That's neither of the two things I previously accused you of conflating. It's a third thing.

Having fully general counterarguments against your hated enemies, and lots of blog posts readying your troops for battle with those hated enemies, is not generally a winning way to avoid getting into lots of messy time-wasting fights. If one of LW's top catchphrases is 'boo politics!', we'll thereby by setting ourselves up as the Anti-Politics Tribe, a hated enemy of the Politics Tribes. The Politics Tribes are precisely the people we're trying to avoid picking fights with, especially not fights framed as tribalistic no-holds-barred absolutist sloganeering shouting matches.

Compare "We can't be against war in the Middle East, or else the Middle-Eastern-War-Fighting-Tribe will recognize us as their hated enemy and destroy us." This is not how it works. The Israelis dislike the Palestinians. The Palestinians dislike the Israelis. There is not a Middle-Eastern-War-Fighting-Tribe, composed of Israelis and Palestinians in equal parts, which values war in the Middle East as a terminal value and coordinates to defend it against its detractors.

There is no Politics Tribe who get offended by criticizing politics. There are various political groups who get offended if you allow politics and then some tiny subcomponent of you associates with the wrong side.

I've previously speculated that tribalism is so inescapable that the only way to have any hope of working towards correct beliefs rather than tribal signaling is founding a tribe around epistemic virtue. As such, I think you're right that we might sort of be starting an Anti-Politics Tribe, insofar as epistemic virtue and standard partisan politics don't mix. But I don't think anyone is going to start identifying as the Anti-Epistemic-Virtue Tribe to oppose us.

That's not my experience, but if that's true, then a lot of the people I'm interested in building ties to are in that high-value has-a-nuanced-position minority.

Is it fair for me to describe your goal as trying to shift our self-presentation to appeal to highly-political people?

I think we can both agree that we shouldn't exclude anyone a priori based on their meta-level beliefs about politics.

But I am also getting the impression that you think highly-political people are especially high value, whereas I think they are especially low value.

Consider the situation of a meetup group in a sketchy part of town. Occasionally there is gang violence nearby, but the meetup group is made up of nice people and has thus far mostly avoided it.

A member of the group has a bright idea. "Let's try especially hard to recruit hardened gangsters to our group. After all, they are extremely knowledgeable in gang violence and can protect us if any violence comes our way. At the very least, they can tell us from a position of experience what we should do to minimize our risk."

There is some truth to that argument.

But there's the counterargument that having lots of hardened gangsters in a group might make it a much more likely target for gang violence, and that inviting them in puts everyone at much greater risk.

More important, there's another counterargument that hardened gangsters are often violent people, and even if they don't provoke conflicts with gangsters outside the group, the next time the group has an argument about what kind of soda to bring to the meetup they might find that being full of hardened gangsters from opposing gangs makes it really hard to solve problems peacefully and cooperatively.

I think importing a lot of political people is likely to have the same dynamics - increased threat of violence from outside, increased threat of conflict from within. We already dodged a huge from-outside-bullet when most of the neoreactionaries moved over to More Right and Eliezer very publicly denied having any idea what they were talking about, thus denying Slate the "weird technolibertarian nerds probably in bed with crazy racists" article we both know they would have loved to write. And we already had to ban Eugene - a man interested in politics if ever there was one - for causing internal strife in a way that took years to detect and resolve and probably drove away a lot of good people. Do we really want to select our recruitment efforts for people with the same risk profile?

Comment author: RobbBB 22 July 2014 05:32:54AM *  5 points [-]

One way of beginning to address that problem might be to use 'X is hard mode' as a schema for a lot of other things people have trouble talking about. It can also be agent-relative; 'sorry, eugenics is hard mode for me' is a nice stock-phrase alternative to 'sorry, I find eugenics triggering' (which might be objectionable if the 'trigger' isn't literally a PTSD trigger; or, if it is a PTSD trigger, the sufferer might not want to divulge that much). If the group of common 'hard modes' includes a lot of things that are more intuitively unsavory than 'politics', there's less bravado risk.

Though I don't think bravado is a very large concern, here and now and in practice. (At least compared to the kinds of things I tend to worry about re group norms.) Regarding politics, your comment is more cynical than my post; but regarding memes and conversation, I think my post is the more cynical. Neither of us trusts people to talk politics well, but I also don't trust people to talk about not-talking-about-politics well. So I suggested a meme that I think is useful, but also can fail gracefully in normal, everyday usage.

LW is not at risk anytime soon of falling in love with politics, but it is at risk of appearing arrogant, dismissive, insulting, thoughtlessly-opposed-to-local-politics-and-groupcraft, etc. Its most widely used memes should probably be useful for outreach and for teaching new ideas -- people get exposed to LW through sloppy, improvised conversations, at least as much as through carefully crafted blog posts -- not just useful for making known facts more introspectively salient to people who already know all the fundamentals. (Of course, it'd be nice if something served both purposes.)

I can see 'politics is the mind-killer' being fairly useful to a fan of Dune who's familiar with the heuristics and biases literature (especially confirmation bias, sophistication effects, motivated reasoning, bias blind spots), has read some of your and Eliezer's writing about politics, and is entering the discussion feeling extremely friendly, sympathetic, and non-defensive. That's why I mentioned that 'politics is the mind-killer' might be a useful internal mantra, like 'politics is SPIDERS'.

But I don't think spiders or mind-kill is a useful meme for people who haven't internalized the sequences. (Including people on LW; 22% of people who took the last LW survey said they'd read at most ~25% of the sequences.) 'Mindkill' is likelier to make people authentically mad than authentically scared; if it makes them scared it probably won't be for the right reasons; and when I run mental simulations of chatroom or meet-up conversations, I have an easy time envisioning a conversation that's starting to go off the rails staying unproductive, or getting worse, because someone said 'you guys are getting mind-killed!' or 'psh, don't you know that politics is the mind-killer!'. (And people who've actually overheard the term seem often to agree with my impression.)

Maybe we aren't disagreeing bigly, but are primarily intuiting different typical usage scenarios for stock LW-isms? Mental mantra and Secret Word of Power, v. something you throw out in mixed company or use to convey new information to someone. I envision "politics is hard mode" serving a social role more than a mnemonic one, to guide conversation and show newbies the ropes. It's OK if it doesn't try to encode an explicit warning 'ALSO HARD MODE IS A LOT HARDER THAN YOU THINK, AND PEOPLE CAN DIE, AND AHHHHH', because the way the mantra is being used (to pivot the group away from a political discussion) conveys that information, plus people can optionally talk about all that stuff when it's strategic to do so, but eschew bringing that point up if all they really want to convey is 'politics is epistemically hard; let's do something easier'. Cutting memes up into their atomic sub-ideas is useful for mental focus, but also for incremental pedagogy.

It's the sort of thing that could work on someone who hasn't heard the idea 'politics is the mind-killer' before, or has heard it but hasn't fully understood it, or understands it but doesn't fully agree with or accept it. I'm interested in talking to at least some of those people.

Comment author: Yvain 22 July 2014 05:00:45PM *  37 points [-]

LW is not at risk anytime soon of falling in love with politics, but it is at risk of appearing arrogant, dismissive, insulting, thoughtlessly-opposed-to-local-politics-and-groupcraft, etc.

This might be the crux of our disagreement.

I don't have statistics for Less Wrong, but here are some for SSC. The topic is "median number of page views for different types of post throughout 2014".

As you can see, interest in charity and statistics is the lowest, followed by interest in transhumanism and rationality. Politics is the highest of the group that clusters around the 3000s. Then comes "race and gender" at 8000, and "things i will regret writing" (my tag for very controversial political rants that will make a lot of people very angry) at 16000, ie about five times the level for rationality or transhumanism.

This seems to correspond to how things work on Less Wrong, where for example a basic introduction of misogyny and mansplaining got almost twice as many comments as Anna's massive and brilliant post resolving a bunch of philosophy of mind issues and more than three times as many as Luke's heavily researched primer on fighting procrastination.

Not to mention that disaster with Eugene was politically based. I'm pretty sure nobody mass-downvotes because someone else disagrees with them about GiveWell.

Less Wrong is massively at risk of falling in love with politics. Politics is much more interesting and attention-sucking than working on important foundational questions, and as soon as we relax the taboo on it we are doomed. On the other hand, most of the people who say we're "arrogant" will find a reason to think so no matter how we phrase things. I mean, what happens when they're okay with our pithy slogan on politics, look at the site, and figure out what we actually believe?

That having been said, if you've been doing a lot of public relations work and empirically find a lot of people are turned off by the way "politics is the mind-killer" is used in practice, I can't tell you you're wrong. I just hope that however you choose to push the same idea doesn't result in a sudden influx of people who think politics is great and are anxious to prove they're capable of "hard mode".

Comment author: Yvain 22 July 2014 04:21:27AM *  48 points [-]

"Hard mode" sounds too metal. The proper response to "X is hard mode" is "Bring it on!"

Therefore I object to "politics is hard mode" for the same reason I object to "driving a car with your eyes closed is hard mode". Both statements are true, but phrased to produce maximum damage.

There's also a way that "politics is hard mode" is worse than playing a video game on hard mode, or driving a car on hard mode. If you play the video game and fail, you know and you can switch back to an easier setting. If you drive a car in "hard mode" and crash into a tree, you know you should keep your eyes open the next time.

If you discuss politics in "hard mode", you can go your entire life being totally mind-killed (yes! I said it!) and just think everyone else is wrong, doing more and more damage each time you open your mouth and destroying every community you come in contact with.

Can you imagine a human being saying "I'm sorry, I'm too low-level to participate in this discussion"? There may be a tiny handful of people wise enough to try it - and ironically, those are probably the same handful who have a tiny chance of navigating the minefield. Everyone else is just going to say "No, I'm high-enough level, YOU'RE the one who needs to bow out!"

Both "hard mode" and "mind-killer" are intended to convey a sense of danger, but the first conveys a fun, exciting danger that cool people should engage with as much as possible in order to prove their worth, and the latter conveys an extreme danger that can ruin everything and which not only clouds your faculties but clouds the faculty to realize that your faculties are clouded. As such, I think "mind-killer" is the better phrase.

EDIT: More succintly: both phrases mean the same thing, but with different connotations. "Hard mode" sounds like we should accord more status to politics, "mind-killer" sounds like we should accord less. I feel like incentivizing more politics is a bad idea and will justify this if anyone disagrees.

Comment author: Yvain 07 July 2014 01:17:16AM *  23 points [-]

This is a very good metaphor and I approve of you making it.

But it only works when there are a reasonably small number of predecessors.

If there have been a thousand gladiators before you, fine.

If there have been a trillion gladiators before you, someone else has had this idea, and that makes the absence of names on the coliseum ten times more scary.

If there have been a trillion gladiators before you, the conditions have been in place for gladiators to bribe others to put their names on the wall since the beginning of gladiating, and there are still no names on the wall, then you are fundamentally misunderstanding some aspect of the situation. Either people are lying when they say there have been a trillion gladiators before you, or people are lying on a much more fundamental level - for example, the walls of the Coliseum are wiped clear once a year, or this entire scenario is a dream.

If we assume a late filter and set up this scenario, the obvious question becomes "What happened to the last civilization who tried this?"

And the obvious answer is "nothing good".

Since it seems unlikely that every past civilization collapsed by coincidence just before it could implement this idea, then with certain strong assumptions like "very many civilizations" and "ease of galaxy-wide transmission", we are left with only the possibility of early filter, or careful enforcement of late filter by alien intelligence.

The reason that this scenario requires such nonsensical decision theory is because it's based on flawed assumptions - that this state of affairs plus a late filter could ever come about naturally.

I hope this seems like a logical development of what I said in the post you linked.

Comment author: Yvain 03 June 2014 02:28:09PM *  3 points [-]

Yesterday I posted a Michigan meetup.

My location is set to Michigan.

The "Nearest Meetups" column on the right-hand side suggests Atlanta and Houston, but not Michigan.

Is this a known bug?

In response to Against Open Threads
Comment author: James_Miller 30 May 2014 06:43:21PM 25 points [-]

Lesswrong has degraded because high-quality and previously prolific authors including Eliezer, Luke, and Yvain write much less for us. One quick way to improve the quality of LW would be for Yvain to cross-post from his personal blog here (which I would particularly benefit from since his blog doesn't let me comment and I'm almost certain this isn't because Yvain banned me.)

Comment author: Yvain 31 May 2014 10:37:23AM *  15 points [-]

Possibly the ban on James Donald affected everyone named James. Try posting again now.

Comment author: NancyLebovitz 23 May 2014 03:59:08PM 3 points [-]

Slatestarcodex isn't loading for me. It's obviously loading for other people-- I'm getting email notifications of comments. I use chrome.

Anyone have any idea what the problem might be?

Comment author: Yvain 24 May 2014 02:48:38AM *  2 points [-]

It wasn't working for me either all day. A few hours ago it mysteriously reappeared. I called tech support. They said they had no explanation.

It should be up again now. I will investigate better hosting solutions.

Comment author: torekp 19 March 2014 02:14:47AM 0 points [-]

Will there be chit-chat after training? Rebecca wants to come along iff so.

Comment author: Yvain 20 March 2014 04:14:37AM 0 points [-]

Yes, at the very least we'll go out to dinner.

Comment author: ArisKatsaris 18 March 2014 10:08:26PM 6 points [-]

What happened to slatestarcodex and does anyone know if it's just temporary or something to be concerned about?

Comment author: Yvain 19 March 2014 12:19:42AM 18 points [-]

My hosting company got annoyed because something was taking up too many resources. I did what the nice person on the telephone suggested (installed some WordPress plugins, uninstalled others) and it's back online now. If the problem recurs I might have to restrict commenting for a while until I can figure out a more permanent solution, but for now everything's fine.

Meetup : Detroit/Ann Arbor - Memory Workshop

4 Yvain 17 March 2014 11:45PM

Discussion article for the meetup : Detroit/Ann Arbor - Memory Workshop

WHEN: 23 March 2014 01:00:00PM (-0400)

WHERE: 19334 Angling Street, Livonia, MI

Brienne from CFAR will be coming all the way from California to teach a couple hour workshop on memory and mnemonic techniques with some application to productivity hacking as well. Same location as usual. Donations appreciated.

Additional special guest Robby Bensinger of http://nothingismere.com/ (user: RobbBB)

Discussion article for the meetup : Detroit/Ann Arbor - Memory Workshop

Comment author: ChrisHallquist 04 March 2014 02:08:18AM 0 points [-]

But then we expect mainstream academia to be wrong in a lot of cases - you bring up the case of mainstream academic philosophy, and although I'm less certain than you are there, I admit I am very skeptical of them.

With philosophy, I think the easiest, most important thing for non-experts to notice is that (with a few arguable exceptions are independently pretty reasonable) philosophers basically don't agree on anything. In the case of e.g. Plantinga specifically, non-experts can notice few other philosophers think the modal ontological argument accomplishes anything.

The crackpot warning signs are good (although it's interesting how often basically correct people end up displaying some of them because they get angry at having their ideas rejected and so start acting out...

Examples?

We are so fascinated by the man-bites-dog cases of very intelligent people believing stupid things that it's hard to remember that stupid things are still much, much likelier to be believed by stupid people.

(possible exceptions in politics, but politics is a weird combination of factual and emotive claims, and even the wrong things smart people believe in politics are in my category of "deserve further investigation and charitable treatment".)

I don't think "smart people saying stupid things" reaches anything like man-bites-dog levels of surprisingness. Not only do you have examples from politics, but also from religion. According to a recent study, a little over a third of academics claim that "I know God really exists and I have no doubts about it," which is maybe less than the general public but still a sizeable minority (and the same study found many more academics take some sort of weaker pro-religion stance). And in my experience, even highly respected academics, when they try to defend religion, routinely make juvenile mistakes that make Plantinga look good by comparison. (Remember, I used Plantinga in the OP not because he makes the dumbest mistakes per se but as an example of how bad arguments can signal high intelligence.)

So when I say I judge people by IQ, I think I mean something like what you mean when you say "a track record of making reasonable statements", except basing "reasonable statements" upon "statements that follow proper logical form and make good arguments" rather than ones I agree with.

Proper logical form comes cheap, just add a premise which says, "if everything I've said so far is true, then my conclusion is true." "Good arguments" is much harder to judge, and seems to defeat the purpose of having a heuristic for deciding who to treat charitably: if I say "this guy's arguments are terrible," and you say, "you should read those arguments more charitably," it doesn't do much good for you to defend that claim by saying, "well, he has a track record of making good arguments."

Comment author: Yvain 07 March 2014 08:23:45PM 3 points [-]

I agree that disagreement among philosophers is a red flag that we should be looking for alternative positions.

But again, I don't feel like that's strong enough enough. Nutrition scientists disagree. Politicians and political scientists disagree. Psychologists and social scientists disagree. Now that we know we can be looking for high-quality contrarians in those fields, how do we sort out the high-quality ones from the lower-quality ones?

Examples?

Well, take Barry Marshall. Became convinced that ulcers were caused by a stomach bacterium (he was right; later won the Nobel Prize). No one listened to him. He said that "my results were disputed and disbelieved, not on the basis of science but because they simply could not be true...if I was right, then treatment for ulcer disease would be revolutionized. It would be simple, cheap and it would be a cure. It seemed to me that for the sake of patients this research had to be fast tracked. The sense of urgency and frustration with the medical community was partly due to my disposition and age."

So Marshall decided since he couldn't get anyone to fund a study, he would study it on himself, drank a serum of bacteria, and got really sick.

Then due to a weird chain of events, his results ended up being published in the Star, a tabloid newspaper that by his own admission "talked about alien babies being adopted by Nancy Reagan", before they made it into legitimate medical journals.

I feel like it would be pretty easy to check off a bunch of boxes on any given crackpot index..."believes the establishment is ignoring him because of their biases", "believes his discovery will instantly solve a centuries-old problem with no side effects", "does his studies on himself", "studies get published in tabloid rather than journal", but these were just things he naturally felt or had to do because the establishment wouldn't take him seriously and he couldn't do things "right".

I don't think "smart people saying stupid things" reaches anything like man-bites-dog levels of surprisingness. Not only do you have examples from politics, but also from religion. According to a recent study, a little over a third of academics claim that "I know God really exists and I have no doubts about it," which is maybe less than the general public but still a sizeable minority

I think it is much much less than the general public, but I don't think that has as much to do with IQ per se as with academic culture. But although I agree that the finding that IQ isn't a stronger predictor of correct beliefs than it is is interesting, I am still very surprised that you don't seem to think it matters at all (or at least significantly). What if we switched gears? Agreeing that the fact that a contrarian theory is invented or held by high IQ people is no guarantee of its success, can we agree that the fact that a contrarian theory is invented and mostly held by low IQ people is a very strong strike against it?

Proper logical form comes cheap, just add a premise which says, "if everything I've said so far is true, then my conclusion is true."

Proper logical form comes cheap, but a surprising number of people don't bother even with that. Do you frequently see people appending "if everything I've said so far is true, then my conclusion is true" to screw with people who judge arguments based on proper logical form?

Comment author: ChrisHallquist 03 March 2014 06:09:51PM *  1 point [-]

I question how objective these objective criterion you're talking about are. Usually when we judge someone's intelligence, we aren't actually looking at the results of an IQ test, so that's subjective. Ditto rationality. And if you were really that concerned about education, you'd stop paying so much attention to Eliezer or people who have a bachelors' degree at best and pay more attention to mainstream academics who actually have PhDs.

FWIW, actual heuristics I use to determine who's worth paying attention to are

  • What I know of an individual's track record of saying reasonable things.
  • Status of them and their ideas within mainstream academia (but because everyone knows about this heuristic, you have to watch out for people faking it.
  • Looking for other crackpot warning signs I've picked up over time, e.g. a non-expert claiming the mainstream academic view is not just wrong but obviously stupid, or being more interested in complaining that their views are being suppressed than in arguing for those views.

Which may not be great heuristics, but I'll wager that they're better than IQ (wager, in this case, being a figure of speech, because I don't actually know how you'd adjudicate that bet).

It may be helpful, here, to quote what I hope will be henceforth known as the Litany of Hermione: "The thing that people forget sometimes, is that even though appearances can be misleading, they're usually not."

You've also succeeded in giving me second thoughts about being signed up for cryonics, on the grounds that I failed to consider how it might encourage terrible mental habits in others. For the record, it strikes me as quite possible that mainstream neuroscientists are entirely correct to be dismissive of cryonics—my biggest problem is that I'm fuzzy on what exactly they think about cryonics (more here).

Comment author: Yvain 03 March 2014 07:12:26PM 9 points [-]

Your heuristics are, in my opinion, too conservative or not strong enough.

Track record of saying reasonable things once again seems to put the burden of decision on your subjective feelings and so rule out paying attention to people you disagree with. If you're a creationist, you can rule out paying attention to Richard Dawkins, because if he's wrong about God existing, about the age of the Earth, and about homosexuality being okay, how can you ever expect him to be right about evolution? If you're anti-transhumanism, you can rule out cryonicists because they tend to say lots of other unreasonable things like that computers will be smarter than humans, or that there can be "intelligence explosions", or that you can upload a human brain.

Status within mainstream academia is a really good heuristic, and this is part of what I mean when I say I use education as a heuristic. Certainly to a first approximation, before investigating a field, you should just automatically believe everything the mainstream academics believe. But then we expect mainstream academia to be wrong in a lot of cases - you bring up the case of mainstream academic philosophy, and although I'm less certain than you are there, I admit I am very skeptical of them. So when we say we need heuristics to find ideas to pay attention to, I'm assuming we've already started by assuming mainstream academia is always right, and we're looking for which challenges to them we should pay attention to. I agree that "challenges the academics themselves take seriously" is a good first step, but I'm not sure that would suffice to discover the critique of mainstream philosophy. And it's very little help at all in fields like politics.

The crackpot warning signs are good (although it's interesting how often basically correct people end up displaying some of them because they get angry at having their ideas rejected and so start acting out, and it also seems like people have a bad habit of being very sensitive to crackpot warning signs the opposing side displays and very obtuse to those their own side displays). But once again, these signs are woefully inadequate. Plantinga doesn't look a bit like a crackpot.

You point out that "Even though appearances can be misleading, they're usually not." I would agree, but suggest you extend this to IQ and rationality. We are so fascinated by the man-bites-dog cases of very intelligent people believing stupid things that it's hard to remember that stupid things are still much, much likelier to be believed by stupid people.

(possible exceptions in politics, but politics is a weird combination of factual and emotive claims, and even the wrong things smart people believe in politics are in my category of "deserve further investigation and charitable treatment".)

You are right that I rarely have the results of an IQ test (or Stanovich's rationality test) in front of me. So when I say I judge people by IQ, I think I mean something like what you mean when you say "a track record of making reasonable statements", except basing "reasonable statements" upon "statements that follow proper logical form and make good arguments" rather than ones I agree with.

So I think it is likely that we both use a basket of heuristics that include education, academic status, estimation of intelligence, estimation of rationality, past track record, crackpot warning signs, and probably some others.

I'm not sure whether we place different emphases on those, or whether we're using about the same basket but still managing to come to different conclusions due to one or both of us being biased.

Comment author: ChrisHallquist 03 March 2014 12:42:28AM *  5 points [-]

So although I would endorse Aumann-adjusting as a final verdict with many of the people on this site, I think it's great that we have discussions - even heated discussions - first, and I think a lot of those discussions might look from the outside like disrespect and refusal to Aumann adjust.

I agree that what look like disrespectful discussions at first could eventually lead to Aumann agreement, but my impression is that there are a lot of persistent disagreements within the online rationalist community. Eliezer's disagreements with Robin Hanson are well-known. My impression is that even people within MIRI have persistent disagreements with each other, though not as big as the Eliezer-Robin disagreements. I don't know for sure Alicorn and I would continue to disagree about the ethics of white lies if we talked it out thoroughly, but it wouldn't remotely surprise me. Et cetera.

The role that IQ is playing here is that of a quasi-objective Outside View measure of a person's ability to be correct and rational. It is, of course, a very very lossy measure that often goes horribly wrong. On the other hand, it makes a useful counterbalance to our subjective measure of "I feel I'm definitely right; this other person has nothing to teach me."

So we have two opposite failure modes to avoid here. The first failure mode is the one where we fetishize the specific IQ number even when our own rationality tells us something is wrong - like Plantiga being apparently a very smart individual, but his arguments being terribly flawed. The second failure mode is the one where we're too confident in our own instincts, even when the numbers tell us the people on the other side are smarter than we are. For example, a creationist says "I'm sure that creationism is true, and it doesn't matter whether really fancy scientists who use big words tell me it isn't."

I guess I need to clarify that I think IQ is a terrible proxy for rationality, that the correlation is weak at best. And your suggested heuristic will do nothing to stop high IQ crackpots from ignoring the mainstream scientific consensus. Or even low IQ crackpots who can find high IQ crackpots to support them. This is actually a thing that happens with some creationists—people thinking "because I'm an <engineer / physicist / MD / mathematician>, I can see those evolutionary biologists are talking nonsense." Creationists would do better to attend to the domain expertise of evolutionary biologists. (See also: my post on the statistician's fallacy.)

I'm also curious as to how much of your willingness to agree with me in dismissing Plantinga is based on him being just one person. Would you be more inclined to take a sizeable online community of Plantingas seriously?

Unless you are way way way more charitable than I am, I have a hard time believing that you are anywhere near the territory where the advice "be less charitable" is more helpful than the advice "be more charitable".

As I said above, you can try to pinpoint where to apply this advice. You don't need to be charitable to really stupid people with no knowledge of a field. But once you've determined someone is in a reference class where there's a high prior on them having good ideas - they're smart, well-educated, have a basic committment to rationality - advising that someone be less charitable to these people seems a lot like advising people to eat more and exercise less - it might be useful in a couple of extreme cases, but I really doubt it's where the gain for the average person lies.

On the one hand, I dislike the rhetoric of charity as I see it happen on LessWrong. On the other hand, in practice, you're probably right that people aren't too charitable. In practice, the problem is selective charity—a specific kind of selective charity, slanted towards favoring people's in-group. And you seem to endorse this selective charity.

I've already said why I don't think high IQ is super-relevant to deciding who you should read charitably. Overall education also doesn't strike me as super-relevant either. In the US, better educated Republicans are more likely to deny global warming and think that Obama's a Muslim. That appears to be because (a) you can get a college degree without ever taking a class on climate science and (b) more educated conservatives are more likely to know what they're "supposed" to believe about certain issues. Of course, when someone has a Ph.D. in a relevant field, I'd agree that you should be more inclined to assume they're not saying anything stupid about that field (though even that presumption is weakened if they're saying something that would be controversial among their peers).

As for "basic commitment to rationality," I'm not sure what you mean by that. I don't know how I'd turn it into a useful criterion, aside from defining it to mean people I'd trust for other reasons (e.g. endorsing standard attitudes of mainstream academia). It's quite easy for even creationists to declare their commitment to rationality. On the other hand, if you think someone's membership in the online rationalist community is a strong reason to treat what they say charitably, yeah, I'm calling that self-congratulatory nonsense.

And that's the essence of my reply to your point #5. It's not people having self-congratulatory attitudes on an individual level. It's the self-congratulatory attitudes towards their in-group.

Comment author: Yvain 03 March 2014 04:22:06PM *  10 points [-]

I agree that what look like disrespectful discussions at first could eventually lead to Aumann agreement, but my impression is that there are a lot of persistent disagreements within the online rationalist community. Eliezer's disagreements with Robin Hanson are well-known. My impression is that even people within MIRI have persistent disagreements with each other, though not as big as the Eliezer-Robin disagreements. I don't know for sure Alicorn and I would continue to disagree about the ethics of white lies if we talked it out thoroughly, but it wouldn't remotely surprise me. Et cetera.

Are ethics supposed to be Aumann-agreeable? I'm not at all sure the original proof extends that far. If it doesn't, that would cover your disagreement with Alicorn as well as a very large number of other disagreements here.

I don't think it would cover Eliezer vs. Robin, but I'm uncertain how "real" that disagreement is. If you forced both of them to come up with probability estimates for an em scenario vs. a foom scenario, then showed them both each other's estimates and put a gun to their heads and asked them whether they wanted to Aumann-update or not, I'm not sure they wouldn't agree to do so.

Even if they did, it might be consistent with their current actions: if there's a 20% chance of ems and 20% chance of foom (plus 60% chance of unpredictable future, cishuman future, or extinction) we would still need intellectuals and organizations planning specifically for each option, the same way I'm sure the Cold War Era US had different branches planning for a nuclear attack by USSR and a nonnuclear attack by USSR.

I will agree that there are some genuinely Aumann-incompatible disagreements on here, but I bet it's fewer than we think.

I guess I need to clarify that I think IQ is a terrible proxy for rationality, that the correlation is weak at best. And your suggested heuristic will do nothing to stop high IQ crackpots from ignoring the mainstream scientific consensus. Or even low IQ crackpots who can find high IQ crackpots to support them.

So I want to agree with you, but there's this big and undeniable problem we have and I'm curious how you think we should solve it if not through something resembling IQ.

You agree people need to be more charitable, at least toward out-group members. And this would presumably involve taking people whom we are tempted to dismiss, and instead not dismissing them and studying them further. But we can't do this for everyone - most people who look like crackpots are crackpots. There are very likely people who look like crackpots but are actually very smart out there (the cryonicists seem to be one group we can both agree on) and we need a way to find so we can pay more attention to them.

We can't use our subjective feeling of is-this-guy-a-crackpot-or-not, because that's what got us into this problem in the first place. Presumably we should use the Outside View. But it's not obvious what we should be Outside Viewing on. The two most obvious candidates are "IQ" and "rationality", which when applied tend to produce IQ fetishism and in group favoritism (since until Stanovich actually produces his rationality quotient test and gives it to everybody, being in a self-identified rationalist community and probably having read the whole long set of sequences on rationality training is one of the few proxies for rationality we've got available).

I admit both of these proxies are terrible. But they seem to be the main thing keeping us from, on the one side, auto-rejecting all arguments that don't sound subjectively plausible to us at first glance, and on the other, having to deal with every stupid creationist and homeopath who wants to bloviate at us.

There seems to be something that we do do that's useful in this sphere. Like if someone with a site written in ALL CAPS and size 20 font claims that Alzheimers is caused by a bacterium, I dismiss it without a second thought because we all know it's a neurodegenerative disease. But a friend who has no medical training but whom I know is smart and reasonable recently made this claim, I looked it up, and sure enough there's a small but respectable community of microbiologists and neuroscientists investigating that maybe Alzheimers is triggered by an autoimmune response to some bacterium. It's still a long shot, but it's definitely not crackpottish. So somehow I seem to have some sort of ability for using the source of an implausible claim to determine whether I investigate it further, and I'm not sure how to describe the basis on which I make this decision beyond "IQ, rationality, and education".

I'm also curious as to how much of your willingness to agree with me in dismissing Plantinga is based on him being just one person. Would you be more inclined to take a sizeable online community of Plantingas seriously?

Well, empirically I did try to investigate natural law theology based on there being a sizeable community of smart people who thought it was valuable. I couldn't find anything of use in it, but I think it was a good decision to at least double-check.

On the one hand, I dislike the rhetoric of charity as I see it happen on LessWrong. On the other hand, in practice, you're probably right that people aren't too charitable. In practice, the problem is selective charity—a specific kind of selective charity, slanted towards favoring people's in-group. And you seem to endorse this selective charity.

If you think people are too uncharitable in general, but also that we're selectively charitable to the in-group, is that equivalent to saying the real problem is that we're not charitable enough to the out-group? If so, what subsection of the out-group would you recommend we be more charitable towards? And if we're not supposed to select that subsection based on their intelligence, rationality, education, etc, how do we select them?

And if we're not supposed to be selective, how do we avoid spending all our time responding to total, obvious crackpots like creationists and Time Cube Guy?

On the other hand, if you think someone's membership in the online rationalist community is a strong reason to treat what they say charitably, yeah, I'm calling that self-congratulatory nonsense. And that's the essence of my reply to your point #5. It's not people having self-congratulatory attitudes on an individual level. It's the self-congratulatory attitudes towards their in-group.

Yeah, this seems like the point we're disagreeing on. Granted that all proxies will be at least mostly terrible, do you agree that we do need some characteristics that point us to people worth treating charitably? And since you don't like mine, which ones are you recommending?

Comment author: Yvain 02 March 2014 07:50:21AM *  34 points [-]

I interpret you as making the following criticisms:

1. People disagree with each other, rather than use Aumann agreement, which proves we don't really believe we're rational

Aside from Wei's comment, I think we also need to keep track of what we're doing.

If we were to choose a specific empirical fact or prediction - like "Russia will invade Ukraine tomorrow" - and everyone on Less Wrong were to go on Prediction Book and make their prediction and we took the average - then I would happily trust that number more than I would trust my own judgment. This is true across a wide variety of different facts.

But this doesn't preclude discussion. Aumann agreement is a way of forcing results if forcing results were our only goal, but we can learn more by trying to disentangle our reasoning processes. Some advantages to talking about things rather than immediately jumping to Aumann:

  • We can both increase our understanding of the issue.

  • We may find a subtler position we can both agree on. If I say "California is hot" and you say "California is cold", instead of immediately jumping to "50% probability either way" we can work out which parts of California are hot versus cold at which parts of the year.

  • We may trace part of our disagreement back to differing moral values. If I say "capital punishment is good" and you say "capital punishment is bad", then it may be right for me to adjust a little in your favor since you may have evidence that many death row inmates are innocent, but I may also find that most of the force of your argument is just that you think killing people is never okay. Depending on how you feel about moral facts and moral uncertainty, we might not want to Aumann adjust this one. Nearly everything in politics depends on moral differences at least a little.

  • We may trace our disagreement back to complicated issues of worldview and categorization. I am starting to interpret most liberal-conservative issues as a tendency to draw Schelling fences in different places and then correctly reason with the categories you've got. I'm not sure if you can Aumann-adjust that away, but you definitely can't do it without first realizing it's there, which takes some discussion.

So although I would endorse Aumann-adjusting as a final verdict with many of the people on this site, I think it's great that we have discussions - even heated discussions - first, and I think a lot of those discussions might look from the outside like disrespect and refusal to Aumann adjust.

2. It is possible that high IQ people can be very wrong and even in a sense "stupidly" wrong, and we don't acknowledge this enough.

I totally agree this is possible.

The role that IQ is playing here is that of a quasi-objective Outside View measure of a person's ability to be correct and rational. It is, of course, a very very lossy measure that often goes horribly wrong. On the other hand, it makes a useful counterbalance to our subjective measure of "I feel I'm definitely right; this other person has nothing to teach me."

So we have two opposite failure modes to avoid here. The first failure mode is the one where we fetishize the specific IQ number even when our own rationality tells us something is wrong - like Plantiga being apparently a very smart individual, but his arguments being terribly flawed. The second failure mode is the one where we're too confident in our own instincts, even when the numbers tell us the people on the other side are smarter than we are. For example, a creationist says "I'm sure that creationism is true, and it doesn't matter whether really fancy scientists who use big words tell me it isn't."

We end up in a kind of bravery debate situation here, where we have to decide whether it's worth warning people more against the first failure mode (at the risk it will increase the second), or against the second failure mode more (at the risk that it will increase the first).

And, well, studies pretty universally find everyone is overconfident of their own opinions. Even the Less Wrong survey finds people here to be really overconfident.

So I think it's more important to warn people to be less confident they are right about things. The inevitable response is "What about creationism?!" to which the counterresponse is "Okay, but creationists are stupid, be less confident when you disagree with people as smart or smarter than you."

This gets misinterpreted as IQ fetishism, but I think it's more of a desperate search for something, anything to fetishize other than our own subjective feelings of certainty.

3. People are too willing to be charitable to other people's arguments.

This is another case where I think we're making the right tradeoff.

Once again there are two possible failure modes. First, you could be too charitable, and waste a lot of time engaging with people who are really stupid, trying to figure out a smart meaning to what they're saying. Second, you could be not charitable enough by prematurely dismissing an opponent without attempting to understand her, and so perhaps missing out on a subtler argument that proves she was right and you were wrong all along.

Once again, everyone is overconfident. No one is underconfident. People tell me I am too charitable all the time, and yet I constantly find I am being not-charitable-enough, unfairly misinterpreting other people's points, and so missing or ignoring very strong arguments. Unless you are way way way more charitable than I am, I have a hard time believing that you are anywhere near the territory where the advice "be less charitable" is more helpful than the advice "be more charitable".

As I said above, you can try to pinpoint where to apply this advice. You don't need to be charitable to really stupid people with no knowledge of a field. But once you've determined someone is in a reference class where there's a high prior on them having good ideas - they're smart, well-educated, have a basic committment to rationality - advising that someone be less charitable to these people seems a lot like advising people to eat more and exercise less - it might be useful in a couple of extreme cases, but I really doubt it's where the gain for the average person lies.

In fact, it's hard for me to square your observation that we still have strong disagreements with your claim that we're too charitable. At least one side is getting things wrong. Shouldn't they be trying to pay a lot more attention to the other side's arguments?

I feel like utter terror is underrated as an epistemic strategy. Unless you are some kind of freakish mutant, you are overconfident about nearly everything and have managed to build up very very strong memetic immunity to arguments that are trying to correct this. Charity is the proper response to this, and I don't think anybody does it enough.

4. People use too much jargon.

Yeah, probably.

There are probably many cases in which the jargony terms have subtly different meaning or serve as reminders of a more formal theory and so are useful ("metacontrarian" versus "showoff", for example), but probably a lot of cases where people could drop the jargon without cost.

I think this is a more general problem of people being bad at writing - "utilize" vs. "use" and all that.

5. People are too self-congratulatory and should be humbler

What's weird is that when I read this post, you keep saying people are too self-congratulatory, but to me it sounds more like you're arguing people are being too modest, and not self-congratulatory enough.

When people try to replace their own subjective analysis of who can easily be dismissed ("They don't agree with me; screw them") with something based more on IQ or credentials, they're being commendably modest ("As far as I can tell, this person is saying something dumb, but since I am often wrong, I should try to take the Outside View by looking at somewhat objective indicators of idea quality.")

And when people try to use the Principle of Charity, once again they are being commendably modest ("This person's arguments seem stupid to me, but maybe I am biased or a bad interpreter. Let me try again to make sure.")

I agree that it is an extraordinary claim to believe anyone is a perfect rationalists. That's why people need to keep these kinds of safeguards in place as saving throws against their inevitable failures.

Comment author: Eliezer_Yudkowsky 28 February 2014 08:09:35PM 2 points [-]

CPAP (auto-adjusting pressure) didn't work on me. What else is there?

Comment author: Yvain 01 March 2014 01:11:00AM 3 points [-]

Complicated. I think I'm seeing you tomorrow night, I'll talk to you then rather than demand your medical history on a public forum.

Comment author: James_Miller 25 February 2014 08:04:38PM 5 points [-]

To illustrate dead-weight loss in my intro micro class I first take out a dollar bill and give it to a student and then explain that the sum of the wealth of the people in the classroom hasn't changed. Next, I take a second dollar bill and rip it up and throw it in the garbage. My students always laugh nervously as if I've done something scandalous like pulling down my pants. Why?

Comment author: Yvain 28 February 2014 09:47:53AM 7 points [-]

Because it signals "I am so wealthy that I can afford to tear up money" and blatantly signaling wealth is crass. And it also signals "I am so callous that I would rather tear up money than give it to the poor", which is also crass. And the argument that a one dollar bill really isn't very much money isn't enough to disrupt the signal.

Comment author: James_Miller 24 February 2014 07:15:11PM 5 points [-]

What if you are Jewish and are trying to stop a Hitler from coming to power and the best means would be to spread a deliberate lie about him. Are you saying that the worse the outcome would be, the less likely you would be to lie?

Comment author: Yvain 28 February 2014 09:35:40AM *  2 points [-]

I think the relevant axis may be short-term/specific vs. long-term/broader consequences rather than unimportant vs. important. I think defecting is usually a long-term bad strategy but a short-term good one. If you're pretty sure there's not going to be a long-term unless you fix your short-term problems immediately, defecting might be a good idea for you or your chosen cause - not sure about for the world at large.

Comment author: Yvain 28 February 2014 07:06:26AM *  25 points [-]

This is good stuff!

One addition I would make to your "sleep" section: between 5% and 10% of Americans have moderate or severe sleep apnea, mostly undiagnosed. Untreated sleep apnea more than doubles mortality through a combination of cardiac problems, stroke, and maybe a cancer-promoting effect as well. There are well-known effective treatments for sleep apnea and it is kind of dumb not to get them.

The main symptoms of sleep apnea are excessive snoring, and feeling very tired during the day even if you slept a normal amount the night before. It is most common in obese and older people but sometimes happens in normal-weight and younger people as well. If you think you might have this condition, probably your highest-priority longevity intervention (after quitting smoking, if you do that) is to go to your doctor and get it checked out.

Comment author: Brillyant 21 February 2014 05:17:43AM *  4 points [-]

Are you willing to agree to a statement like:

"Weight loss is possible in some cases, and in fact very easy in some cases. In other cases it is very hard, bordering in impossible given the marathon-analogy definition of impossible below. This can be negated by heroic measures like locking them in a room where excess food is unavailable and ignoring their appetite and distress, but in the real world you cannot do this. Because of these difficult cases, it is useful to explore the science behind weight loss and come up with more effective strategies."

Yes, basically. Though I don't know if I like the inclusion of "real" world. It isn't possible to deprive people of excess calories in the practical world, but calories math and weight loss are, as far as I know, a physical law in the real universe.

If so, we agree, but then I'm confused why you were criticizing the Less Wrongers in your original statement.

Since this thread became longer and more involved than I thought, I should go back and find a few examples from the discussion I mention in my original post.

Generally, the discussion seemed to be over-complicating a simple issue. Forgive me if I mentioned this to you already, but I see two distinct discussions in regard to weight loss:

  • 1 - The causal mechanisms that lead to losing weight.

  • 2 - Executing a rational plan with those causal mechanisms in mind.

In my view (I'll look for examples), the issues were conflated, with 1 receiving the majority of debate. I don't think 1 is in need of debate, and I now wonder if people were just looking for a way to hack the causal mechanisms because dieting via 2 is super hard.

I sincerely empathize. With depression, I feel sometimes like people try to suggest all sorts of 2-styled help to me, for instance. They sort of come across like, "Just choose these different behaviors and it will alleviate your depression". In that way, they are telling folks with depression they believe it to be a choice.

Even though I hate it, they are kind of correct. I can't currently do anything to change the causal mechanisms that lead to obesity or depression. I'd like to, but I can't. I gotta do 2 to feel happy. If I can't, I'm gonna be sad, no matter much I pretend I can change 1.

I do think there are hacks for 2. And I'm 100% in favor of exploring them. As I said before, I used many tricks to help my will and motivation while dieting.

But if you want to lose weight, you have to create a calorie deficit. As you go on to say, the body will used stored calories if you don't eat enough. Living that calorie deficit (dieting) can blow. It blew for me. But it was a trade off in order that I could accomplish the task of losing weight. It was necessary to lose weight.

There seems to be a notion with some here at LW that experiencing some hunger or fatigue or less-than-perfect-happy-functionality is a dieting failure mode. It isn't. It is like anything else in life. To your analogy, you don't run faster times by rationalizing about why you can't run faster times. It DOES hurt a bit—even when it is doing no damage to your body—to push through the rationalization. Of course, you can push too hard—in running or in dieting—but I'm not talking about exceeding what is healthy.

To sum up, somebody replied to me (somewhere on here) with an EY quote that said: "I can starve or I can think. I can't do both" I assume it was in the context of dieting.

I think(?) it provides a good summary of the attitude I'm criticizing on LW in regard to dieting. You don't have to starve to diet. But there may be no way in the physical universe to avoid losing some % of your comfort and/or capability during your diet (calorie deficit). You may lose some "think" as you do a little bit of "starve". Just like you lose a little comfort when you do a little bit faster run... and I lose a little time and sleep when I type a little more. :)

I'll look for examples.

Comment author: Yvain 21 February 2014 06:51:30AM *  10 points [-]

I agree that trying to avoid all pain can be a failure mode. But insisting that pain needs to be plowed through can also be a failure mode.

The advice "You should run a marathon by continuing to run even if it hurts" might perhaps be useful as part of a package of different interventions to a runner who's hit some kind of a motivational wall.

But in other situations it is completely inappropriate. For example, suppose a certain runner has a broken leg, but you don't know this and he can't communicate it to you. He just says "It really really hurts when I run!" And you just answer "Well, you need to run through the pain!"

This is an unreasonable request. If you were more clueful, you might make a suggestion like "You should go to a doctor, wait for your broken leg to heal, and then try running later."

And if enough people have broken legs, then promoting the advice "You should run a marathon by continuing to run even when it hurts" is bad advice. Even if we assume that people are still capable of running on broken legs and will not collapse, you are generalizing from your own example to assume that the pain they suffer will be minimal and tolerable, rather than excruciating and intolerable.

If some people have metabolic problems - and right now I'm not claiming they do, just creating a possibility proof, and if you agree it's possible but don't think it's real we can get into that later - then they're like the broken legged people.

Working off Taubes and a few other of the low-carb people, some people's fat cells do not release energy. If they suffer any caloric deficit at all, even 300 calories or whatever you consider a reasonable small amount, their bodies will immediately start starving and catabolizing muscles or, in the worst case, vital organs. There have been examples (though the implications and generalizability are still debated) of people starving to death while weighing three or four hundred pounds, because for some reason their bodies couldn't get to their fat and so were forced to catabolize the liver or heart or something.

(this is what "starving to death" tends to mean in real life; you are so starved that your body breaks down important tissues it can't afford to lose, or builds up too many tissue-breakdown waste products)

Imagine that you are literally starving to death - let's say you've been without food for three months and you're down to the bone like those heart-wrenching photos out of some African countries. You have no energy and can barely move out of your bed. In theory it is possible for you to use willpower to force yourself to continue going on with your daily activities, and not have complaints like "I can't starve and think at the same time". In practice, this seems like a pretty poor plan if you have any other options.

If Taubes et al are correct, fat people who can't mobilize their body's energy reserves are in exactly this situation. Any caloric deficit and they're literally starving, their bodies are trying to figure out whether they should cannibalize the heart or the liver first, and they're not in the mood for continuing to go about daily activities with a smile on their face any more than that African in the famine is.

If your body has fully functional fat metabolism, and you're operating at a caloric deficit by successfully burning off fat, and you tell them "Hey, I feel moderately hungry but really this isn't so hard", you're comparing apples to oranges, the same way as the healthy runner to the runner with the broken leg.

A better solution would be to come up with some way to fix the thing where the body can't mobilize its fat reserves and so either has to stay fat or starve to death. I think this is the project the paleo people are working on: figuring out how to make the body say "Okay, caloric deficit, better burn some fat cells" instead of "Oh no, caloric deficit, better assume I'm starving to death and jack appetite up to eleven while catabolizing all of my muscles".

So is our disagreement that you don't think even Taubes' picture provides a situation in which one should privilege bodyhacking-type solutions over willpower-based solutions, or just that you don't think Taubes' picture is correct?

Comment author: Brillyant 21 February 2014 03:23:36AM 3 points [-]

But by your theory, they should still be able to go to work, maintain all their relationships, and otherwise behave exactly like a non-depressed person in every way. In practice this seems very hard for depressed people and a lot of the burden of depression is effects from not being able to do this.

I think you missed my point, or I threw it by you poorly. I don't think they "should", I think they sometimes can. I sometimes can, and though I know from LW that not all minds are alike, it's safe to assume I'm also not wholly unique in my depression.

As you go on to point out, there is some baseline threshold for which people cannot will themselves out of depression or other psychological issues, just as there are weight loss diets and exercise programs they cannot succeed at.

To your analogy of the marathon: There is a right answer to whether you can run X miles in Y minutes and not physically injure yourself. I'd imagine the majority of people never come close to knowing that answer because they do not posses the ability to refrain from rationalizing themselves out of the optimal result as discomfort begins during their run. I'm aware that I'm personally really bad at this—my first .5 mile I'm telling myself I'm Usain Bolt; by mile 2 I'm coming up with manifold reasons to stop pushing. No doubt some are good and rational reasons, but others are bullshit that I need only train my mind to recognize as such in order to push through and be successful.

The answer to your second question is that their body would become upset because it's not getting the calories it needs...etc...

That's not really what I was asking. Maybe I asked poorly.

Can you instead imagine a scenario where a controlled calorie deficit was administered to a person where they received a balanced diet with all the nutrients they needed? 300 kcal was arbitrary. Pick any number. Isn't there some number that would be negligible in terms of the utility of its nutritional value and yet provide a calories deficient sufficient to lead to weight loss?

My point was that depression seems to have no such scenario. You cannot engineer a situation to "make" people be happy. Without excess food, or alcohol, you cannot get to obesity or alcohol addiction, right? Depression has no such outside variable.

Comment author: Yvain 21 February 2014 04:17:41AM *  8 points [-]

I think you missed my point, or I threw it by you poorly. I don't think they "should", I think they sometimes can. I sometimes can, and though I know from LW that not all minds are alike, it's safe to assume I'm also not wholly unique in my depression.

I agree that they sometimes can. I also agree people can sometimes lose weight. As far as I was concerned, our disagreement here (if one exists) isn't about whether it's possible in some cases.

Are you willing to agree to a statement like:

"Weight loss is possible in some cases, and in fact very easy in some cases. In other cases it is very hard, bordering in impossible given the marathon-analogy definition of impossible below. This can be negated by heroic measures like locking them in a room where excess food is unavailable and ignoring their appetite and distress, but in the real world you cannot do this. Because of these difficult cases, it is useful to explore the science behind weight loss and come up with more effective strategies.

If so, we agree, but then I'm confused why you were criticizing the Less Wrongers in your original statement. If you don't agree, please let me know which part we disagree about.

To your analogy of the marathon: There is a right answer to whether you can run X miles in Y minutes and not physically injure yourself. I'd imagine the majority of people never come close to knowing that answer because they do not posses the ability to refrain from rationalizing themselves out of the optimal result as discomfort begins during their run. I'm aware that I'm personally really bad at this—my first .5 mile I'm telling myself I'm Usain Bolt; by mile 2 I'm coming up with manifold reasons to stop pushing. No doubt some are good and rational reasons, but others are bullshit that I need only train my mind to recognize as such in order to push through and be successful.

If we are debating the extremely academic point of whether someone with your muscular structure can complete a marathon in X hours, okay. But suppose we find that of a thousand people who in theory are anatomically capable of completing the marathon, zero actually finish the marathon, due to discomfort. If our goal is to get them to successfully complete marathons, what percent of our resources do you think should be invested in proving they are physically capable of doing so right now and exhorting them to do this, versus coming up with things like training schedules and better diets and better shoes that will make it easier for them?

I felt like your original point was a complaint that we are trying the equivalent of coming up with training schedules rather than the equivalent of telling people they should be able to just keep going 26.2 miles unless their legs collapse, whereas I think this is probably a better strategy. Am I interpreting your complaint correctly, and do you disagree that the former strategy is better?

Can you instead imagine a scenario where a controlled calorie deficit was administered to a person where they received a balanced diet with all the nutrients they needed? 300 kcal was arbitrary. Pick any number. Isn't there some number that would be negligible in terms of the utility of its nutritional value and yet provide a calories deficient sufficient to lead to weight loss?

I think we're definitely misunderstanding each other somewhere. I think we may be working off some different assumptions about how the biology here works.

I weigh 185 pounds - plugging this into a metabolism calculator, my weight will stay stable at 2200 calories per day. Suppose I weighed 500 pounds. My weight would stay stable at about 4500 calories per day.

If the 500 pound guy got only 4200 calories per day, it doesn't matter how balanced the diet is or how many nutrients he has, his body has a caloric deficit and doesn't have enough energy to live. Hopefully it takes care of that by burning some of his stored fat. If it can't do that, he's going to be in big trouble.

I may be wrong about this, but I don't think the body can actually operate a a true caloric deficit. It WILL make up the deficit (or die, which also technically resolves the deficit). All it can do is do so in more or less problematic ways. The less problematic ways are things like burning fat. The more problematic ways are things like increasing appetite, decreasing exercise, and catabolizing organs.

I think your question corresponds to "But what if the body did just operate at a caloric deficit?", and I am really getting out of my knowledge comfort zone here but I don't think that's possible. Our analogy to economics here fails - we're not talking money where you can run a loss for a while and just have to worry about the bank coming after you, we're talking thermodynamics where it's physically impossible.

Weight loss is caused not by operating at a caloric deficit per se, but by the body avoiding caloric deficit by burning fat or other bodily tissues.

I could be totally wrong about this.

Comment author: Brillyant 20 February 2014 04:22:04PM 1 point [-]

I like the analogy, because I can personally relate to depression. I don't know that it is a great one, though.

We know very well how to cause weight loss. It's a calorie deficit issue, and we could force it to occur. That is, we can will weight loss.

It has not been my experience that people can will happiness—not even for a short time. They can (sometimes) will themselves to be productive, and smile, and go to work, and even drudge through exercise. But willing happiness is not a possibility I am aware of.

It isn't my argument that we should "force" weight loss, only that we can. We should be as sciency as we can be in order to come up with more convenient and reasonable ways to help be lose weight. I gamified it. I used some LW-ish principles.

Questions:

  • I assume you do not consider depression a choice. That is, depressed people cannot chose to become undepressed. They may choose to engage in behaviors that alleviate depression, but certain people are so severely depressed that they cannot summon the will to even engage in the depression-alleviate behaviors. Is this an accurate summary?

  • If someone's caloric balance were 100% controlled so that they had a 300 kcal daily deficit, what would happen to that persons weight over the course of 30 days? 90 days? 1 year? What would happen to their appetite? Metabolism? BMI? Assume they are given a careful balance adequate nutrients. Assume they are given freedom to exercise and be active to their heart's content. An exact 300 kcal deficit is alwasy 100% enforced. What would result?

Comment author: Yvain 21 February 2014 02:43:06AM *  13 points [-]

I think you've rescued the rule that depressive people can't just decide to feel happy. But by your theory, they should still be able to go to work, maintain all their relationships, and otherwise behave exactly like a non-depressed person in every way. In practice this seems very hard for depressed people and a lot of the burden of depression is effects from not being able to do this. The metaphor that just as this is a hard problem and worthy of scientific attention, so weight loss can be a hard problem and worthy of scientific attention still holds.

But why stick with depression? I could just as easily move to obsessive-compulsive disorder. Can't they just "force" not washing their hands too often? Or social phobia - can't they just "force" themselves to go out and socialize when appropriate?

Probably the best example is substance abuse - can't people just "force" themselves not to drink alcohol? And yet not only do therapy-type interventions like Alcoholics Anonymous appear to work, but purely biological interventions like Vivitrol seem to work as well. I am pretty happy that these exist and the more of them people can think up for weight loss, the better.

It isn't my argument that we should "force" weight loss, only that we can.

I didn't interpret your original point that way. You said "There seemed to be all sorts of discussion about everything other than the simple math behind weight loss. Lots of super fascinating stuff—but much of it missing the point, I thought...they seem to spin their wheels so badly on a discussion about something as simple as weight loss" It sounded to me like you had negative opinions about the tendency to discuss non-forcing strategies for weight loss. Am I misinterpreting?

But my main objection here would be the word "can". This word is useful in everyday speech but horrible in subtle philosophical discussions about willpower because it imports a series of assumptions that are exactly what we should be trying to discuss.

It is written: "It's easy to run a marathon. All you have to do is start running, and not stop until you've gone 26.2 miles." As far as I know this could be correct - whenever I've stopped running before reaching a goal, it hasn't been because my body has literally collapsed, it's been because I felt really tired and uncomfortable and so decided to stop. I guess it's possible that if I could ignore that, my body would literally shut down before the 26.2 mark, but I've never been able to get that far and my bet is neither have you.

So is it true that I "can" run a marathon but I just don't "want to"? My guess is that a lot of how inability works is that when your body is getting upset about something, it makes doing that thing more and more unpleasant until doing it passes beyond anyone's conceivable pain/willpower threshold and that person stops. If that's true, then looking at things in terms of "could have kept running" is going to totally fail to capture what's going on.

This answers your first question.

The answer to your second question is that their body would become upset because it's not getting the calories it needs. It might respond by limiting physical activity, either by making the person involved so tired that they don't exercise as much as they used to, and thus cutting their caloric expenditure by 300. It might decrease invisible metabolic things to make up for some of the deficit, like making the person fidget less and decreasing body temperature. Between these two things it might be able to balance its caloric budget again.

If that didn't happen, in healthy people where everything is working properly it would start making adipose tissue release fat to make up the shortfall (I am going to assume these people's diets are perfectly balanced other than the caloric deficit). I have heard many smart people claim that in some people, this process is deranged, adipose tissue does not release fat effectively, and the body would be forced to go to its backup plan of cannibalizing muscle and vital organs, which over long periods is not compatible with life. I have not investigated this thoroughly enough to see if it is true. In either case they would lose weight.

So by the end of [time period], my current best understanding is that the subjects would either be the same weight, lower weight, or dead, depending on whose theories are correct, what diet they were put on, individual differences, and what the time period was. Sorry I can't be more specific.

Comment author: palladias 19 February 2014 03:48:49PM 37 points [-]

A simple reframe that helped jumpstart my creativity:

My cookie dough froze in the fridge, so I couldn't pry it out of the bowl to carry with me to bake at a party. I tried to get it out, but didn't succeed, and had basically resigned myself to schlepping the bowl on the metro.

But then I paused and posed the question to myself: "If something important depended on me getting this dough out, what would I try?"

I immediately covered the top of the bowl, ran the base under lukewarm to warm water, popped it out, wrapped it up, and went on my way.

Comment author: Yvain 20 February 2014 08:15:30PM *  26 points [-]

After reading the third paragraph, I had already decided to post the following similar story:

It snowed a few weeks ago and my car was stuck in the driveway. Parts of the wheels had gotten ice/snow kind of frozen/compacted around them. I was breaking up the ice with one of those things you use to break up ice, but a lot of it was too hard and a lot of it was underneath the car and I couldn't get to it. I was pretty close to being late to work. So I thought "I need to make some kind of desperate rationalist effort here, what would HPJEV do?". And I sat and thought about it for five minutes, and I got a big tub, filled it with hot water, and poured it around the wheels. This melted/softened enough of the compacted ice that I was able to break up the rest and make it to work on time.

Then I read your fourth paragraph and saw your story was also about hot water.

I don't know if there's some kind of moral to this episode, like that the most rational solution to a problem always involves hot water, but I guess I'll raise it a little higher on my list of things to think about in various situations.

Comment author: Brillyant 19 February 2014 05:37:52PM 18 points [-]

I've lost 30 pounds since September 17th, 2013*. Interestingly, I've noticed doing so caused me to lose a lot of faith in LW.

In the midst of my diet, discussion in the comments on this series of posts confounded me. I'm no expert on nutrition or dieting(I do know perhaps more than the average person), but my sense is that I encountered a higher noise-to-signal ratio on the subject here at LW than anywhere else I've looked. There seemed to be all sorts of discussion about everything other than the simple math behind weight loss. Lots of super fascinating stuff—but much of it missing the point, I thought.

I learned a few interesting things during the discussion—which I always seem to do here. But in terms of providing a boost to my instrumental rationality, it didn't help at all. In fact, it's possible LW had a negative impact on my ability to win at dieting and weight management.

I notice this got me wondering about LW's views and discussions about many other things that I know very little about. I feel myself asking "How could I rationally believe LW knows what they are talking about in regard to the Singularity, UFAI, etc. if they seem to spin their wheels so badly on a discussion about something as simple as weight loss?"

I'm interested to hear others' thoughts on this.

Have you ever lost confidence in LW after a similar experience? Maybe something where it seemed to you people were "talking a big game" but failing to apply any of that to actually win in real life?

(*Note: To be clear, I've lost 30 pounds since Sept 17th, but only ~15-18 lbs since my "diet" began on Jan 1, 2014. I'm not really bragging about losing weight—I wish it weren't the case. I injured my neck and could no longer use my primary method of exercise (weightlifting) to stay in shape. After eating poorly and lying around for a couple months, I started—on Jan 1—to do consistent, light treadmill work & light core work, as well as cutting my calorie consumption pretty dramatically.)

Comment author: Yvain 19 February 2014 10:36:02PM *  28 points [-]

Consider the following story:

I was feeling a little blue. I looked at the psychiatric literature, and they were saying all this weird stuff about neurotrophic factors and cognitive-behavioral therapy. But then that night I had dinner with some friends, went to the gym for an hour, and sure enough I felt a lot better afterwards!

I would have at least three qualms with such an attitude:

First, there are different kinds of low mood. Some differences are obvious; some people are less depressed than others, or depressed for much shorter time periods. But it could also be that there are no visible differences between two people, but that for hidden reasons one person's depression will respond to some quick exercise and social activity, and another person's won't.

Second, even interventions that are known to always work can be hard to task-ify. Exercise is indeed often a very effective treatment for depression, but when you tell a depressed person "just go and exercise", they usually won't do that because they're too depressed. Having a good social support network can be helpful in depression, but depressed people can be unable to make friends because deep down they assume everyone hates them. Part of treating depression is bringing people to the point where they're able to do the simple interventions. If you get a depressed person who does have the motivation to exercise and make friends, great, but it's not a point against psychiatry that they sometimes discuss how to help people who don't.

A third problem is general anti-scientificness. Yeah, sure, you don't need to understand exactly how neurogenesis occurs in order to treat depression. But it's neat to know. And in fact exercise may treat depression by increasing neurotrophic factors, so you're not disagreeing with the scientists, just looking at it from a different angle. And for certain people it might be, in a weird way, sort of inspirational to know the science and help them figure out why they're doing what they're doing. If they want to study it, why complain?

I think most of the same issues generalize to your comment.

I would also add one more, which is that it is generally much easier to lose weight on a diet than to keep the weight off for more than a year or two. For example, of people who lost (hey, look at that!) thirty pounds on a diet, one year later they had on average gained back fifteen of them. Longer followups usually find even more of the weight regained; see for example Mann 2007. So you're declaring something simple before you've even started the hard part.

Comment author: Leonhart 16 February 2014 07:55:44PM 7 points [-]

Brought to mind by the recent post about dreaming on Slate Star Codex:

Has anyone read a convincing refutation of the deflationary hypothesis about dreams - that is, that there aren't any? In the sense of nothing like waking experience ever happening during sleep; just junk memories with backdated time-stamps?

My brain is attributing this position to Dennett in one of his older collections - maybe Brainstorms - but it probably predates him.

Comment author: Yvain 16 February 2014 11:30:46PM *  18 points [-]

Stimuli can be incorporated into dreams - for example, if someone in a sleep lab sees you are in REM sleep and sprays water on you, you're more likely to report having had a dream it was raining when you wake up. Yes, this has been formally tested. This provides strong evidence that dreams are going on during sleep.

More directly, communication has been established between dreaming and waking states by lucid dreamers in sleep labs. Lucid dreamers can make eye movements during their dreams to send predetermined messages to laboratory technicians monitoring them with EEGs. Again, this has been formally tested.

Comment author: VAuroch 04 February 2014 11:59:12PM -1 points [-]

This seemed clever for a minute. Then I remembered that East and West don't even generalize to a global scale, let alone an extraplanetary one, and it stopped seeming clever.

Comment author: Yvain 07 February 2014 05:48:15AM 4 points [-]

One can talk about points on Earth corresponding to the position of a celestial body - for example, the high tide being "directly under" the moon, or noon being "directly under" the sun.

If it is noon in California and high tide in New York, and you're in Missouri, I think it makes sense to say you are east of the sun, west of the moon.

Comment author: Yvain 02 February 2014 03:56:26PM *  3 points [-]

'East of the Sun, West of the Moon,' rather than being an unreachable fairy tale place, actually refers to where I am like 25% of the time.

-- @superlativeish

Comment author: hyporational 30 January 2014 06:54:56AM *  0 points [-]

A small investigational drug trial won't be powered to detect outliers, and you won't be able to reliably solve that by invoking Bayesian statistics.

I think in the hypothetical he meant you've already won the lottery, so to speak.

The whole "medical doctors can always consistently treat medical diseases, but psychiatrists are throwing darts blindfolded" story is something of a myth

I agree, too bad for the patients who actually need help that the myth is alive and well. Psychiatry allows for this blind folded dart throwing more though since there are no simple tests, and people might be judging the whole field based on a few incompetent individuals or psychotherapy forms that have stuck for historical reasons. I don't think you can directly compare medications to make the point like they did in that paper, since drugs make up a smaller fraction of psychiatrists' treatment arsenal. Correct me if it's different in the US.

(Take psychodynamic psychotherapy for example and see how popular it is for whatever reason. I doubt you'll find such a popular rotten corpse in medicine.) I was wrong about this one apparently, thanks Yvain. If you do, I suppose it would be some surgical technique. Both psychotherapy and surgery require training so there are greater sunk costs involved.

Comment author: Yvain 31 January 2014 05:43:37AM 4 points [-]

Psychotherapy seems to work pretty well, and it's not obvious that psychodynamic psychotherapy works less well than other sorts. See http://slatestarcodex.com/2013/09/19/scientific-freud/ . I prefer things more in the CBT vein myself, but the pro-psychodynamics people aren't as helpless and discredited as one might think.

Comment author: Yvain 30 January 2014 03:40:11AM *  12 points [-]

A small investigational drug trial won't be powered to detect outliers, and you won't be able to reliably solve that by invoking Bayesian statistics.

In large drug trials I think this is to some degree already done, but it's limited by the extreme sketchiness of suddenly inventing new endpoints for your study after you have the data. It would probably take the form of increasing the threshold for an endpoint (for example, "No significant difference between drug and placebo was found with the planned endpoint of decreasing HAM-D ratings by 3 or more, but there were significantly more patients in the drug group who had their HAM-D ratings decrease by 10 or more". Everyone is rightly suspicious of people who do this, because, again, changing endpoints. But if it happened enough someone would take notice. Trust me, "not coming up with clever ways to make their drug look effective for at least some people" is not one of pharmaceutical companies' failure modes.

But keep in mind that you sort of loaded the original example by choosing something that almost never happens (someone living to 110 without any signs of aging). In a psychiatry study, what's the most extreme example you're going to get? Someone's depression remits completely? Big deal. Most people's depressive episodes remit completely after a couple of months anyway, and in 25% of people they never return (in even more people, they take many years to return, and almost no studies continue for the many years it would take to notice). In a drug trial of 10000 people (the number you gave above) hundreds or thousands of people in each group are going to have their depression remit completely; if the drug has a superpowerful effect on one person and cures her depression forever, that will get lost in noise in the way that someone living to 110 with the body of a 30 year old might not.

(it's instructive to compare this to the way studies investigate side effects. If one person in a 10000 person study has their arms fall off, the investigators will notice, because that's sufficiently rare as to raise suspicion it was caused by the drug. The drug will then end up with a black box warning saying "may make arms fall off.")

Another way these sorts of outlier effects might be detected is by subgroup analyses (which are also extremely sketchy). If there is no effect in general, researchers may check whether there is an effect among men, among women, among blacks, among whites, among Latinos, among postmenopausal Burmese women who wear hats and own at least two pets and have a history of disease in their left kidney, anything that turns up a positive result. But again, this is hardly something we want to encourage.

But all these things are for investigational drugs. if we're talking about a drug that's already been approved and has a strong prescription history, then your worries about individualized response would get subsumed into the good responder / bad responder distinction, which is a very very big area of research which we know a lot about and when we don't know it it's not for lack of trying.

For example, among bipolar patients, response to lithium can be (very inconsistently) predicted by selecting for patients who have stronger family history of disease, have fewer depressive symptoms, have slower cycles, have more euthymic periods, have less of a history of drug use, start with a manic episode, demonstrate psychomotor retardation, demonstrate premorbid mood lability, lack premorbid personality disturbance, possibly have deranged serotonin metabolism in platelets, possibly have increased calcium binding to red blood cells, possibly lack the HLA-A3 antigen, possibly have a particular variant of the gene GADL1, etc etc etc.

(in practice we don't expend much effort to check most of these things, because their predictive power is so weak that it's almost always a worse idea than just making a best guess based on the data you have, putting someone on lithium or on an alternative, then switching if it doesn't work)

As far as I know, psychiatrists cannot reliably predict that a given drug will improve a patient's long-term diagnosis, and psychiatrists/psychologists cannot even reliably agree on what condition a patient is manifesting. Mental disorders appear to resist diagnosis and solution, unlike, say, a broken leg or a sucking chest wound.

The whole "medical doctors can always consistently treat medical diseases, but psychiatrists are throwing darts blindfolded" story is something of a myth - see for example Putting the efficacy of psychiatric and general medicine medication into perspective: review of meta-analyses

Comment author: Brillyant 21 January 2014 03:56:04PM *  2 points [-]

This whole article makes a sleight of hand assumption that more rational = more time on LW.

I'm a proto-rationalist by these criteria. I don't see any reason cryonics can't eventually work. I've no interest in it, and I think it is kinda weird.

Some of that weirdness is the typical frozen dead body stuff. But, more than that, I'm weirded out by the immortality-ism that seems to be a big part of (some of) the tenured LW crowd (i.e. rationalists).

I've yet to hear one compelling argument for why hyper-long life = better. The standard answers seems to be "death is obviously bad and the only way you could disagree is because you are biased" and "more years can equal more utilons".

In the case of the former, yeah, death sucks 'cuz it is an end and often involves lots of pain and inconvenience in the run up to it. To the latter, yeah, I get the jist: More utilons = better. Shut up and do math. Okay.

I'm totally on board with getting rid of gratuitous pain and inconvenience that comes with aging. But, as I said, the "I want to live forever! 'cuz that is winning!" thing is just plain weird to me, at least as much so as the frozen body/head bit.

But what could I know... I'm not rational.

Comment author: Yvain 25 January 2014 05:12:20PM 4 points [-]

This whole article makes a sleight of hand assumption that more rational = more time on LW.

Not particularly. If we found that people who spent more time at church are more likely to believe in Jesus, one possible explanation (albeit not proven to be causal) is that going to church makes one believe in Jesus. Likewise, if we find that people who spend more time on Less Wrong are more likely to take a strange idea seriously, one possible (unproven, but reasonable to hypothesize) explanation is that going to Less Wrong makes one more likely to take strange ideas seriously.

Although it's perfectly reasonable not to want to sign up for cryonics (and I haven't signed up myself) the high probability of success but low signup rate among newcomers versus the lower probability of success and higher signup rate among veterans suggests the variable changing is "taking ideas seriously"; this is orthogonal to whether you should or shouldn't want to sign up for cryonics

(unless your claim is that veterans are more anti-deathist than newbies, which would also explain the data and should probably be tested on the next survey. But I think my point the the higher signup rate among veterans does not mean they are more credulous but reflects thought process change still stands)

"Rationalist" here is used to mean "exposed to rationalist ideas", not "is a rationalist person". I realize that's confusing but I don't have better terminology.

Comment author: SaidAchmiz 21 January 2014 06:44:03PM *  1 point [-]

I don't think it does, actually. The following are three distinct scenarios (as pertain to my point #2):

1. Being entirely unaware of what options/possibilities exist in some domain.

Example: I don't do any investing, and so, prior to this thread, had no opinion on index funds whatsoever, nor on mutual funds, nor on anything related.

2. Being unaware of some* particular* (potentially counterintuitive) idea or option.

Example: I'd never had anyone recommend modafinil to me, or suggest that I should take it, or explain what benefits it might have.

3. Being aware of some idea or option, but not taking it seriously.

Example: I have no idea. Gaming poorly-designed lotteries? I suspect this example fails for other reasons, but it does fit the criterion #2.


The claim, as I understand it, was:

There are numerous cases like scenario 3 above, where the main thing that keeps people from taking advantage of an opportunity, and winning thusly, is not taking some idea seriously — despite being aware of that idea. Rationalists, on the other hand, do take the idea seriously, and win thusly.

Index funds are not a good example for people who have no knowledge of investing, because what kept me, for instance, from taking advantage of the profit opportunities offered by the idea "invest in index funds" was not having any knowledge of investing whatsoever, not some failure to take things seriously.

Modafinil is not a good example for people not aware of modafinil or its (alleged) positive effects, because what kept me, for instance, from taking advantage of the cognitive boosts offered by the idea "take modafinil" was not being aware of modafinil, not some failure to take things seriously.

I haven't gotten a good response about Bitcoin, so I won't comment on that.

Now, don't get me wrong: I think index funds are a good example in general, based on the very helpful and clear comments I've gotten on that topic (thank you, commenters!). (Modafinil is not as clearly a good example. I'm still researching.) But my case, and similar others, are not good evidence for those examples.

Comment author: Yvain 25 January 2014 05:06:50PM 1 point [-]

I'm not claiming that a majority of the people who don't do these options don't do them because they're aware of them but don't take them seriously. I'm claiming a majority (or at least many) of the people who possess enough knowledge about them to be able to figure out that they should do them, don't.

My source is mainly anecdotes from people I've talked to who know all the arguments for these but don't do them.

Comment author: SaidAchmiz 21 January 2014 03:14:30AM *  20 points [-]

Yvain, could you give a real-life example analogous to your Goofus & Gallant story?

That is, could you provide an example (or several, even better) of a situation wherein:

  1. There is some opportunity for clear, unambiguous victory;
  2. Taking advantage of it depends primarily on taking a strange/unconventional/etc. idea seriously (as distinct from e.g. not having the necessary resources/connections, being risk-averse, having a different utility function, etc.);
  3. Most people / normal people / non-rationalists do not take the idea seriously, and as a consequence have not taken advantage of said opportunity;
  4. Some people / smart people / rationalists take the idea seriously, and have gone for the opportunity;
  5. And, most importantly, doing so has (not "will"! already has!) caused them to win, in a clear, unambiguous, significant way.

Note that cryonics does not fit that bill (it fails point 5), which is why I'm asking for one or more actual examples.

Comment author: Yvain 21 January 2014 05:09:50AM *  10 points [-]

The example in the thread is real-life-ish - compare to the story of Voltaire and friends winning the French lottery. But if you want more:

It's easy to think of trivial examples of one-time victories - for example, an early Bitcoin investor realizing that crypto-currency had potential and buying some when it was still worth fractions of a cent. But you can justly accuse me of cherry-picking here and demand repeatable examples.

Nothing guarantees that there will be repeatable examples - it could be that people are bad at taking ideas seriously until the ideas succeed once, at which point they realize they were wrong and jump on the bandwagon.

But in fact I think there are such examples. One such is investing in index funds rather than mutual funds/picking your own stocks. There are strong reasons to believe you'll do better, most people know those reasons but don't credit them, and some people do credit them and end up with more money.

Occasional use of modafinil might fall in this category as well, depending on whether we define people's usual reasons for not taking it as irrational or rational-given-different-utility-functions.

I don't think most of these examples will end out as "such obvious wins no one could possibly disagree with them" - with the possible exception of index funds it's never as purely mathematical as the lottery example - but I think for most people the calculus is clear.

Comment author: private_messaging 19 January 2014 02:24:50PM 1 point [-]

The 2012 estimate from SATs was about 128, since the 1994 renorming destroyed the old relationship between the SAT and IQ. Our average SAT (on 1600) was again about 1470, which again maps to less than 130, but not by much. (And, again, self-reported average probably overestimates actual population average.)

It's very interesting that the same mistake was boldly made again this year... I guess this mistake is sort of self reinforcing due to the uncannily perfect equality between mean IQ and what's incorrectly estimated from the SAT scores.

Comment author: Yvain 21 January 2014 03:02:46AM 2 points [-]

According to Vaniver's data downthread, SAT taken only from LWers older than 36 (taking the old SAT) predicts 140 IQ.

I can't calculate the IQ of LWers younger than 36 because I can't find a site I trust to predict IQ from new SAT. The only ones I get give absurd results like average SAT 1491 implies average IQ 151.

Rationalists Are Less Credulous But Better At Taking Ideas Seriously

43 Yvain 21 January 2014 02:18AM

Consider the following commonly-made argument: cryonics is unlikely to work. Trained rationalists are signed up for cryonics at rates much greater than the general population. Therefore, rationalists must be pretty gullible people, and their claims to be good at evaluating evidence must be exaggerations at best.

This argument is wrong, and we can prove it using data from the last two Less Wrong surveys.

The question at hand is whether rationalist training - represented here by extensive familiarity with Less Wrong material - makes people more likely to believe in cryonics.

We investigate with a cross-sectional study, looking at proto-rationalists versus experienced rationalists. Define proto-rationalists as those respondents to the Less Wrong survey who indicate they have been in the community for less than six months and have zero karma (usually indicative of never having posted a comment). And define experienced rationalists as those respondents to the Less Wrong survey who indicate they have been in the community for over two years and have >1000 karma (usually indicative of having written many well-received posts).

By these definitions, there are 93 proto-rationalists, who have been in the community an average of 1.3 months, and 134 experienced rationalists, who have been in the community an average of 4.5 years. Proto-rationalists generally have not read any rationality training material - only 20/93 had read even one-quarter of the Less Wrong Sequences. Experienced rationalists are, well, more experienced: two-thirds of them have read pretty much all the Sequence material.

Proto-rationalists thought that, on average, there was a 21% chance of an average cryonically frozen person being revived in the future. Experienced rationalists thought that, on average, there was a 15% chance of same. The difference was marginally significant (p < 0.1).

Marginal significance is a copout, but this isn't our only data source. Last year, using the same definitions, proto-rationalists assigned a 15% probability to cryonics working, and experienced rationalists assigned a 12% chance. We see the same pattern.

So experienced rationalists are consistently less likely to believe in cryonics than proto-rationalists, and rationalist training probably makes you less likely to believe cryonics will work.

On the other hand, 0% of proto-rationalists had signed up for cryonics compared to 13% of experienced rationalists. 48% of proto-rationalists rejected the idea of signing up for cryonics entirely, compared to only 25% of experienced rationalists. So although rationalists are less likely to believe cryonics will work, they are much more likely to sign up for it. Last year's survey shows the same pattern.

This is not necessarily surprising. It only indicates that experienced rationalists and proto-rationalists treat their beliefs in different ways. Proto-rationalists form a belief, play with it in their heads, and then do whatever they were going to do anyway -  usually some variant on what everyone else does. Experienced rationalists form a belief, examine the consequences, and then act strategically to get what they want.

Imagine a lottery run by an incompetent official who accidentally sets it up so that the average payoff is far more than the average ticket price. For example, maybe the lottery sells only ten $1 tickets, but the jackpot is $1 million, so that each $1 ticket gives you a 10% chance of winning $1 million.

Goofus hears about the lottery and realizes that his expected gain from playing the lottery is $99,999. "Huh," he says, "the numbers say I could actually win money by playing this lottery. What an interesting mathematical curiosity!" Then he goes off and does something else, since everyone knows playing the lottery is what stupid people do.

Gallant hears about the lottery, performs the same calculation, and buys up all ten tickets.

The relevant difference between Goofus and Gallant is not skill at estimating the chances of winning the lottery. We can even change the problem so that Gallant is more aware of the unlikelihood of winning than Goofus - perhaps Goofus mistakenly believes there are only five tickets, and so Gallant's superior knowledge tells him that winning the lottery is even more unlikely than Goofus thinks. Gallant will still play, and Goofus will still pass.

The relevant difference is that Gallant knows how to take ideas seriously.

Taking ideas seriously isn't always smart. If you're the sort of person who falls for proofs that 1 = 2  , then refusing to take ideas seriously is a good way to avoid ending up actually believing that 1 = 2, and a generally excellent life choice.

On the other hand, progress depends on someone somewhere taking a new idea seriously, so it's nice to have people who can do that too. Helping people learn this skill and when to apply it is one goal of the rationalist movement.

In this case it seems to have been successful. Proto-rationalists think there is a 21% chance of a new technology making them immortal - surely an outcome as desirable as any lottery jackpot - consider it an interesting curiosity, and go do something else because only weirdos sign up for cryonics.

Experienced rationalists think there is a lower chance of cryonics working, but some of them decide that even a pretty low chance of immortality sounds pretty good, and act strategically on this belief.

This is not to either attack or defend the policy of assigning a non-negligible probability to cryonics working. This is meant to show only that the difference in cryonics status between proto-rationalists and experienced rationalists is based on meta-level cognitive skills in the latter whose desirability is orthogonal to the object-level question about cryonics.

(an earlier version of this article was posted on my blog last year; I have moved it here now that I have replicated the results with a second survey)

2013 Survey Results

74 Post author: Yvain 19 January 2014 02:51AM

Thanks to everyone who took the 2013 Less Wrong Census/Survey. Extra thanks to Ozy, who helped me out with the data processing and statistics work, and to everyone who suggested questions.

This year's results are below. Some of them may make more sense in the context of the original survey questions, which can be seen here. Please do not try to take the survey as it is over and your results will not be counted.

Part I. Population

1636 people answered the survey.

Compare this to 1195 people last year, and 1090 people the year before that. It would seem the site is growing, but we do have to consider that each survey lasted a different amount of time; for example, last survey lasted 23 days, but this survey lasted 40.

However, almost everyone who takes the survey takes it in the first few weeks it is available. 1506 of the respondents answered within the first 23 days, proving that even if the survey ran the same length as last year's, there would still have been growth.
As we will see lower down, growth is smooth across all categories of users (lurkers, commenters, posters) EXCEPT people who have posted to Main, the number of which remains nearly the same from year to year.

We continue to have very high turnover - only 40% of respondents this year say they also took the survey last year.

II. Categorical Data

SEX:
Female: 161, 9.8%
Male: 1453, 88.8%
Other: 1, 0.1%
Did not answer: 21, 1.3%

[[Ozy is disappointed that we've lost 50% of our intersex readers.]]

GENDER:
F (cisgender): 140, 8.6%
F (transgender MtF): 20, 1.2%
M (cisgender): 1401, 85.6%
M (transgender FtM): 5, 0.3%
Other: 49, 3%
Did not answer: 21, 1.3%

SEXUAL ORIENTATION:
Asexual: 47, 2.9%
Bisexual: 188, 12.2%
Heterosexual: 1287, 78.7%
Homosexual: 45, 2.8%
Other: 39, 2.4%
Did not answer: 19, 1.2%

RELATIONSHIP STYLE:
Prefer monogamous: 829, 50.7%
Prefer polyamorous: 234, 14.3%
Other: 32, 2.0%
Uncertain/no preference: 520, 31.8%
Did not answer: 21, 1.3%

NUMBER OF CURRENT PARTNERS:
0: 797, 48.7%
1: 728, 44.5%
2: 66, 4.0%
3: 21, 1.3%
4: 1, .1%
6: 3, .2%
Did not answer: 20, 1.2%

RELATIONSHIP STATUS:
Married: 304, 18.6%
Relationship: 473, 28.9%
Single: 840, 51.3%

RELATIONSHIP GOALS:
Looking for more relationship partners: 617, 37.7%
Not looking for more relationship partners: 993, 60.7%
Did not answer: 26, 1.6%

HAVE YOU DATED SOMEONE YOU MET THROUGH THE LESS WRONG COMMUNITY?
Yes: 53, 3.3%
I didn't meet them through the community but they're part of the community now: 66, 4.0%
No: 1482, 90.5%
Did not answer: 35, 2.1%

COUNTRY:
United States: 895, 54.7%
United Kingdom: 144, 8.8%
Canada: 107, 6.5%
Australia: 69, 4.2%
Germany: 68, 4.2%
Finland: 35, 2.1%
Russia: 22, 1.3%
New Zealand: 20, 1.2%
Israel: 17, 1.0%
France: 16, 1.0%
Poland: 16, 1.0%

LESS WRONGERS PER CAPITA:
Finland: 1/154,685.
New Zealand: 1/221,650.
Canada: 1/325,981.
Australia: 1/328,659.
United States: 1/350,726
United Kingdom: 1/439,097
Israel: 1/465,176.
Germany: 1/1,204,264.
Poland: 1/2,408,750.
France: 1/4,106,250.
Russia: 1/6,522,727

RACE:
Asian (East Asian): 60, 3.7%
Asian (Indian subcontinent): 37, 2.3%
Black: 11, .7%
Middle Eastern: 9, .6%
White (Hispanic): 73, 4.5%
White (non-Hispanic): 1373, 83.9%
Other: 51, 3.1%
Did not answer: 22, 1.3%

WORK STATUS:
Academics (teaching): 77, 4.7%
For-profit work: 552, 33.7%
Government work: 55, 3.4%
Independently wealthy: 14, .9%
Non-profit work: 46, 2.8%
Self-employed: 103, 6.3%
Student: 661, 40.4%
Unemployed: 105, 6.4%
Did not answer: 23, 1.4%

PROFESSION:
Art: 27, 1.7%
Biology: 26, 1.6%
Business: 44, 2.7%
Computers (AI): 47, 2.9%
Computers (other academic computer science): 107, 6.5%
Computers (practical): 505, 30.9%
Engineering: 128, 7.8%
Finance/economics: 92, 5.6%
Law: 36, 2.2%
Mathematics: 139, 8.5%
Medicine: 31, 1.9%
Neuroscience: 13, .8%
Philosophy: 41, 2.5%
Physics: 92, 5.6%
Psychology: 34, 2.1%
Statistics: 23, 1.4%
Other hard science: 31, 1.9%
Other social science: 43, 2.6%
Other: 139, 8.5%
Did not answer: 38, 2.3%

DEGREE:
None: 84, 5.1%
High school: 444, 27.1%
2 year degree: 68, 4.2%
Bachelor's: 554, 33.9%
Master's: 323, 19.7%
MD/JD/other professional degree: 31, 2.0%
PhD.: 90, 5.5%
Other: 22, 1.3%
Did not answer: 19, 1.2%

POLITICAL:
Communist: 11, .7%
Conservative: 64, 3.9%
Liberal: 580, 35.5%
Libertarian: 437, 26.7%
Socialist: 502, 30.7%
Did not answer: 42, 2.6%

COMPLEX POLITICAL WITH WRITE-IN:
Anarchist: 52, 3.2%
Conservative: 16, 1.0%
Futarchist: 42, 2.6%
Left-libertarian: 142, 8.7%
Liberal: 5
Moderate: 53, 3.2%
Pragmatist: 110, 6.7%
Progressive: 206, 12.6%
Reactionary: 40, 2.4%
Social democrat: 154, 9.5%
Socialist: 135, 8.2%
Did not answer: 26.2%

[[All answers with more than 1% of the Less Wrong population included. Other answers which made Ozy giggle included "are any of you kings?! why do you CARE?!", "Exclusionary: you are entitled to an opinion on nuclear power when you know how much of your power is nuclear", "having-well-founded-opinions-is-really-hard-ist", "kleptocrat", "pirate", and "SPECIAL FUCKING SNOWFLAKE."]]

AMERICAN PARTY AFFILIATION:
Democratic Party: 226, 13.8%
Libertarian Party: 31, 1.9%
Republican Party: 58, 3.5%
Other third party: 19, 1.2%
Not registered: 447, 27.3%
Did not answer or non-American: 856, 52.3%

VOTING:
Yes: 936, 57.2%
No: 450, 27.5%
My country doesn't hold elections: 2, 0.1%
Did not answer: 249, 15.2%

RELIGIOUS VIEWS:
Agnostic: 165, 10.1%
Atheist and not spiritual: 1163, 71.1%
Atheist but spiritual: 132, 8.1%
Deist/pantheist/etc.: 36, 2.2%
Lukewarm theist: 53, 3.2%
Committed theist 64, 3.9%

RELIGIOUS DENOMINATION (IF THEIST):
Buddhist: 22, 1.3%
Christian (Catholic): 44, 2.7%
Christian (Protestant): 56, 3.4%
Jewish: 31, 1.9%
Mixed/Other: 21, 1.3%
Unitarian Universalist or similar: 25, 1.5%

[[This includes all religions with more than 1% of Less Wrongers. Minority religions include Dzogchen, Daoism, various sorts of Paganism, Simulationist, a very confused secular humanist, Kopmist, Discordian, and a Cultus Deorum Romanum practitioner whom Ozy wants to be friends with.]]

FAMILY RELIGION:
Agnostic: 129, 11.6%
Atheist and not spiritual: 225, 13.8%
Atheist but spiritual: 73, 4.5%
Committed theist: 423, 25.9%
Deist/pantheist, etc.: 42, 2.6%
Lukewarm theist: 563, 34.4%
Mixed/other: 97, 5.9%
Did not answer: 24, 1.5%

RELIGIOUS BACKGROUND:
Bahai: 3, 0.2%
Buddhist: 13, .8%
Christian (Catholic): 418, 25.6%
Christian (Mormon): 38, 2.3%
Christian (Protestant): 631, 38.4%
Christian (Quaker): 7, 0.4%
Christian (Unitarian Universalist or similar): 32, 2.0%
Christian (other non-Protestant): 99, 6.1%
Christian (unknown): 3, 0.2%
Eckankar: 1, 0.1%
Hindu: 29, 1.8%
Jewish: 136, 8.3%
Muslim: 12, 0.7%
Native American Spiritualist: 1, 0.1%
Mixed/Other: 85, 5.3%
Sikhism: 1, 0.1%
Traditional Chinese: 11, .7%
Wiccan: 1, 0.1%
None: 8, 0.4%
Did not answer: 107, 6.7%

MORAL VIEWS:
Accept/lean towards consequentialism: 1049, 64.1%
Accept/lean towards deontology: 77, 4.7%
Accept/lean towards virtue ethics: 197, 12.0%
Other/no answer: 276, 16.9%
Did not answer: 37, 2.3%

CHILDREN
0: 1414, 86.4%
1: 77, 4.7%
2: 90, 5.5%
3: 25, 1.5%
4: 7, 0.4%
5: 1, 0.1%
6: 2, 0.1%
Did not answer: 20, 1.2%

MORE CHILDREN:
Have no children, don't want any: 506, 31.3%
Have no children, uncertain if want them: 472, 29.2%
Have no children, want children: 431, 26.7%
Have no children, didn't answer: 5, 0.3%
Have children, don't want more: 124, 7.6%
Have children, uncertain if want more: 25, 1.5%
Have children, want more: 53, 3.2%

HANDEDNESS:
Right: 1256, 76.6%
Left: 145, 9.5%
Ambidextrous: 36, 2.2%
Not sure: 7, 0.4%
Did not answer: 182, 11.1%

LESS WRONG USE:
Lurker (no account): 584, 35.7%
Lurker (account) 221, 13.5%
Poster (comment, no post): 495, 30.3%
Poster (Discussion, not Main): 221, 12.9%
Poster (Main): 103, 6.3%

SEQUENCES:
Never knew they existed: 119, 7.3%
Knew they existed, didn't look at them: 48, 2.9%
~25% of the Sequences: 200, 12.2%
~50% of the Sequences: 271, 16.6%
~75% of the Sequences: 225, 13.8%
All the Sequences: 419, 25.6%
Did not answer: 24, 1.5%

MEETUPS:
No: 1134, 69.3%
Yes, once or a few times: 307, 18.8%
Yes, regularly: 159, 9.7%

HPMOR:
No: 272, 16.6%
Started it, haven't finished: 255, 15.6%
Yes, all of it: 912, 55.7%

CFAR WORKSHOP ATTENDANCE:
Yes, a full workshop: 105, 6.4%
A class but not a full-day workshop: 40, 2.4%
No: 1446, 88.3%
Did not answer: 46, 2.8%

PHYSICAL INTERACTION WITH LW COMMUNITY:
Yes, all the time: 94, 5.7%
Yes, sometimes: 179, 10.9%
No: 1316, 80.4%
Did not answer: 48, 2.9%

VEGETARIAN:
No: 1201, 73.4%
Yes: 213, 13.0%
Did not answer: 223, 13.6%

SPACED REPETITION:
Never heard of them: 363, 22.2%
No,  but I've heard of them: 495, 30.2%
Yes, in the past: 328, 20%
Yes, currently: 219, 13.4%
Did not answer: 232, 14.2%

HAVE YOU TAKEN PREVIOUS INCARNATIONS OF THE LESS WRONG SURVEY?
Yes: 638, 39.0%
No: 784, 47.9%
Did not answer: 215, 13.1%

PRIMARY LANGUAGE:
English: 1009, 67.8%
German: 58, 3.6%
Finnish: 29, 1.8%
Russian: 25, 1.6%
French: 17, 1.0%
Dutch: 16, 1.0%
Did not answer: 15.2%

[[This includes all answers that more than 1% of respondents chose. Other languages include Urdu, both Czech and Slovakian, Latvian, and Love.]]

ENTREPRENEUR:
I don't want to start my own business: 617, 37.7%
I am considering starting my own business: 474, 29.0%
I plan to start my own business: 113, 6.9%
I've already started my own business: 156, 9.5%
Did not answer: 277, 16.9%

EFFECTIVE ALTRUIST:
Yes: 468, 28.6%
No: 883, 53.9%
Did not answer: 286, 17.5%

WHO ARE YOU LIVING WITH?
Alone: 348, 21.3%
With family: 420, 25.7%
With partner/spouse: 400, 24.4%
With roommates: 450, 27.5%
Did not answer: 19, 1.3%

DO YOU GIVE BLOOD?
No: 646, 39.5%
No, only because I'm not allowed: 157, 9.6%
Yes, 609, 37.2%
Did not answer: 225, 13.7%

GLOBAL CATASTROPHIC RISK:
Pandemic (bioengineered): 374, 22.8%
Environmental collapse including global warming: 251, 15.3%
Unfriendly AI: 233, 14.2%
Nuclear war: 210, 12.8%
Pandemic (natural) 145, 8.8%
Economic/political collapse: 175, 1, 10.7%
Asteroid strike: 65, 3.9%
Nanotech/grey goo: 57, 3.5%
Didn't answer: 99, 6.0%

CRYONICS STATUS:
Never thought about it / don't understand it: 69, 4.2%
No, and don't want to: 414, 25.3%
No, still considering: 636, 38.9%
No, would like to: 265, 16.2%
No, would like to, but it's unavailable: 119, 7.3%
Yes: 66, 4.0%
Didn't answer: 68, 4.2%

NEWCOMB'S PROBLEM:
Don't understand/prefer not to answer: 92, 5.6%
Not sure: 103, 6.3%
One box: 1036, 63.3%
Two box: 119, 7.3%
Did not answer: 287, 17.5%

GENOMICS:
Yes: 177, 10.8%
No: 1219, 74.5%
Did not answer: 241, 14.7%

REFERRAL TYPE:
Been here since it started in the Overcoming Bias days: 285, 17.4%
Referred by a friend: 241, 14.7%
Referred by a search engine: 148, 9.0%
Referred by HPMOR: 400, 24.4%
Referred by a link on another blog: 373, 22.8%
Referred by a school course: 1, .1%
Other: 160, 9.8%
Did not answer: 29, 1.9%

REFERRAL SOURCE:
Common Sense Atheism: 33
Slate Star Codex: 20
Hacker News: 18
Reddit: 18
TVTropes: 13
Y Combinator: 11
Gwern: 9
RationalWiki: 8
Marginal Revolution: 7
Unequally Yoked: 6
Armed and Dangerous: 5
Shtetl Optimized: 5
Econlog: 4
StumbleUpon: 4
Yudkowsky.net: 4
Accelerating Future: 3
Stares at the World: 3
xkcd: 3
David Brin: 2
Freethoughtblogs: 2
Felicifia: 2
Givewell: 2
hatrack.com: 2
HPMOR: 2
Patri Friedman: 2
Popehat: 2
Overcoming Bias: 2
Scientiststhesis: 2
Scott Young: 2
Stardestroyer.net: 2
TalkOrigins: 2
Tumblr: 2

[[This includes all sources with  more than one referral; needless to say there was a long tail]]

III. Numeric Data

(in the form mean + stdev (1st quartile, 2nd quartile, 3rd quartile) [n = number responding]))

Age: 27.4 + 8.5 (22, 25, 31) [n = 1558]
Height: 176.6 cm + 16.6 (173, 178, 183) [n = 1267]

Karma Score: 504 + 2085 (0, 0, 100) [n = 1438]
Time in community: 2.62 years + 1.84 (1, 2, 4) [n = 1443]
Time on LW: 13.25 minutes/day + 20.97 (2, 10, 15) [n = 1457]

IQ: 138.2 + 13.6 (130, 138, 145) [n = 506]
SAT out of 1600: 1474 + 114 (1410, 1490, 1560) [n = 411]
SAT out of 2400: 2207 + 161 (2130, 2240, 2330) [n = 333]
ACT out of 36: 32.8 + 2.5 (32, 33, 35) [n = 265]

P(Aliens in observable universe): 74.3 + 32.7 (60, 90, 99) [n = 1496]
P(Aliens in Milky Way): 44.9 + 38.2 (5, 40, 85) [n = 1482]
P(Supernatural): 7.7 + 22 (0E-9, .000055, 1) [n = 1484]
P(God): 9.1 + 22.9 (0E-11, .01, 3) [n = 1490]
P(Religion): 5.6 + 19.6 (0E-11, 0E-11, .5) [n = 1497]
P(Cryonics): 22.8 + 28 (2, 10, 33) [n = 1500]  
P(AntiAgathics): 27.6 + 31.2 (2, 10, 50) [n = 1493]
P(Simulation): 24.1 + 28.9 (1, 10, 50) [n = 1400]
P(ManyWorlds): 50 + 29.8 (25, 50, 75) [n = 1373]
P(Warming): 80.7 + 25.2 (75, 90, 98) [n = 1509]
P(Global catastrophic risk): 72.9 + 25.41 (60, 80, 95) [n = 1502]
Singularity year: 1.67E +11 + 4.089E+12 (2060, 2090, 2150) [n = 1195]

[[Of course, this question was hopelessly screwed up by people who insisted on filling the whole answer field with 9s, or other such nonsense. I went back and eliminated all outliers - answers with more than 4 digits or answers in the past - which changed the results to: 2150 + 226 (2060, 2089, 2150)]]

Yearly Income: $73,226 +423,310 (10,000, 37,000, 80,000) [n = 910]
Yearly Charity: $1181.16 + 6037.77 (0, 50, 400) [n = 1231]
Yearly Charity to MIRI/CFAR: $307.18 + 4205.37 (0, 0, 0) [n = 1191]
Yearly Charity to X-risk (excluding MIRI or CFAR): $6.34 + 55.89 (0, 0, 0) [n = 1150]

Number of Languages: 1.49 + .8 (1, 1, 2) [n = 1345]
Older Siblings: 0.5 + 0.9 (0, 0, 1) [n = 1366]
Time Online/Week: 42.7 hours + 24.8 (25, 40, 60) [n = 1292]
Time Watching TV/Week: 4.2 hours + 5.7 (0, 2, 5) [n = 1316]

[[The next nine questions ask respondents to rate how favorable they are to the political idea or movement above on a scale of 1 to 5, with 1 being "not at all favorable" and 5 being "very favorable". You can see the exact wordings of the questions on the survey.]]

Abortion: 4.4 + 1 (4, 5, 5) [n = 1350]
Immigration: 4.1 + 1 (3, 4, 5) [n = 1322]
Basic Income: 3.8 + 1.2 (3, 4, 5) [n = 1289]
Taxes: 3.1 + 1.3 (2, 3, 4) [n = 1296]
Feminism: 3.8 + 1.2 (3, 4, 5) [n = 1329]
Social Justice: 3.6 + 1.3 (3, 4, 5) [n = 1263]
Minimum Wage: 3.2 + 1.4 (2, 3, 4) [n = 1290]
Great Stagnation: 2.3 + 1 (2, 2, 3) [n = 1273]
Human Biodiversity: 2.7 + 1.2 (2, 3, 4) [n = 1305]

IV. Bivariate Correlations

Ozy ran bivariate correlations between all the numerical data and recorded all correlations that were significant at the .001 level in order to maximize the chance that these are genuine results. The format is variable/variable: Pearson correlation (n). Yvain is not hugely on board with the idea of running correlations between everything and seeing what sticks, but will grudgingly publish the results because of the very high bar for significance (p < .001 on ~800 correlations suggests < 1 spurious result) and because he doesn't want to have to do it himself.

Less Political:
SAT score (1600)/SAT score (2400): .835 (56)
Charity/MIRI and CFAR donations: .730 (1193)
SAT score out of 2400/ACT score: .673 (111)
SAT score out of 1600/ACT score: .544 (102)
Number of children/age: .507 (1607)
P(Cryonics)/P(AntiAgathics): .489 (1515)
SAT score out of 1600/IQ: .369 (173)
MIRI and CFAR donations/XRisk donations: .284 (1178)
Number of children/ACT score: -.279 (269)
Income/charity: .269 (884)
Charity/Xrisk charity: .262 (1161)
P(Cryonics)/P(Simulation): .256 (1419)
P(AntiAgathics)/P(Simulation): .253 (1418)
Number of current partners/age: .238 (1607) 
Number of children/SAT score (2400): -.223 (345)
Number of current partners/number of children: .205 (1612)
SAT score out of 1600/age: -.194 (422)
Charity/age: .175 (1259)
Time on Less Wrong/IQ: -.164 (492)
P(Warming)/P(GlobalCatastrophicRisk): .156 (1522)
Number of current partners/IQ: .155 (521)
P(Simulation)/age: -.153 (1420)
Immigration/P(ManyWorlds): .150 (1195)
Income/age: .150 (930)
P(Cryonics)/age: -.148 (1521)
Income/children: .145 (931)
P(God)/P(Simulation): .142 (1409)
Number of children/P(Aliens): .140 (1523)
P(AntiAgathics)/Hours Online: .138 (1277)
Number of current partners/karma score: .137 (1470)
Abortion/P(ManyWorlds): .122 (1215)
Feminism/Xrisk charity donations: -.122 (1104)
P(AntiAgathics)/P(ManyWorlds) .118 (1381)
P(Cryonics)/P(ManyWorlds): .117 (1387)
Karma score/Great Stagnation: .114 (1202)
Hours online/P(simulation): .114 (1199)
P(Cryonics)/Hours Online: .113 (1279)
P(AntiAgathics)/Great Stagnation: -.111 (1259)
Basic income/hours online: .111 (1200)
P(GlobalCatastrophicRisk)/Great Stagnation: -.110 (1270)
Age/X risk charity donations: .109 (1176)
P(AntiAgathics)/P(GlobalCatastrophicRisk): -.109 (1513)
Time on Less Wrong/age: -.108 (1491)
P(AntiAgathics)/Human Biodiversity: .104 (1286)
Immigration/Hours Online: .104 (1226)
P(Simulation)/P(GlobalCatastrophicRisk): -.103 (1421)
P(Supernatural)/height: -.101 (1232)
P(GlobalCatastrophicRisk)/height: .101 (1249)
Number of children/hours online: -.099 (1321)
P(AntiAgathics)/age: -.097 (1514)
Karma score/time on LW: .096 (1404)

This year for the first time P(Aliens) and P(Aliens2) are entirely uncorrelated with each other. Time in Community, Time on LW, and IQ are not correlated with anything particularly interesting, suggesting all three fail to change people's views.

Results we find amusing: high-IQ and high-karma people have more romantic partners, suggesting that those are attractive traits. There is definitely a Cryonics/Antiagathics/Simulation/Many Worlds cluster of weird beliefs, which younger people and people who spend more time online are slightly more likely to have - weirdly, that cluster seems slightly less likely to believe in global catastrophic risk. Older people and people with more children have more romantic partners (it'd be interesting to see if that holds true for the polyamorous). People who believe in anti-agathics and global catastrophic risk are less likely to believe in a great stagnation (presumably because both of the above rely on inventions). People who spend more time on Less Wrong have lower IQs. Height is, bizarrely, correlated with belief in the supernatural and global catastrophic risk.

All political viewpoints are correlated with each other in pretty much exactly the way one would expect. They are also correlated with one's level of belief in God, the supernatural, and religion. There are minor correlations with some of the beliefs and number of partners (presumably because polyamory), number of children, and number of languages spoken. We are doing terribly at avoiding Blue/Green politics, people.

More Political:
P(Supernatural)/P(God): .736 (1496)
P(Supernatural)/P(Religion): .667 (1492)
Minimum wage/taxes: .649 (1299)
P(God)/P(Religion): .631 (1496)
Feminism/social justice: .619 (1293)
Social justice/minimum wage: .508 (1262)
P(Supernatural)/abortion: -.469 (1309)
Taxes/basic income: .463 (1285)
P(God)/abortion: -.461 (1310)
Social justice/taxes: .456 (1267)
P(Religion)/abortion: -.413
Basic income/minimum wage: .392 (1283)
Feminism/taxes: .391 (1318)
Feminism/minimum wage: .391 (1312)
Feminism/human biodiversity: -.365 (1331)
Immigration/feminism: .355 (1336)
P(Warming)/taxes: .340 (1292)
Basic income/social justice: .311 (1270)
Immigration/social justice: .307 (1275)
P(Warming)/feminism: .294 (1323)
Immigration/human biodiversity: -.292 (1313)
P(Warming)/basic income: .290 (1287)
Social justice/human biodiversity: -.289 (1281)
Basic income/feminism: .284 (1313)
Human biodiversity/minimum wage: -.273 (1293)
P(Warming)/social justice: .271 (1261)
P(Warming)/minimum wage: .262 (1284)
Human biodiversity/taxes: -.251 (1270).
Abortion/feminism: .239 (1356)
Abortion/social justice: .220 (1292)
P(Warming)/immigration: .215 (1315)
Abortion/immigration: .211 (1353)
P(Warming)/abortion: .192 (1340)
Immigration/taxes: .186 (1322)
Basic income/taxes: .174 (1249)
Abortion/taxes: .170 (1328)
Abortion/minimum wage: .169 (1317)
P(warming)/human biodiversity: -.168 (1301)
Abortion/basic income: .168 (1314)
Immigration/Great Stagnation: -.163 (1281)
P(God)/feminism: -.159 (1294)
P(Supernatural)/feminism: -.158 (1292)
Human biodiversity/Great Stagnation: .152 (1287)
Social justice/Great Stagnation: -.135 (1242)
Number of languages/taxes: -.133 (1242)
P(God)/P(Warming): -.132 (1491)
P(Supernatural)/immigration: -.131 (1284)
P(Religion)immigration: -.129 (1296)
P(God)/immigration: -.127 (1286)
P(Supernatural)/P(Warming): -.125 (1487)
P(Supernatural)/social justice: -.125 (1227)
P(God)/taxes: -.145
Minimum wage/Great Stagnation: -124 (1269)
Immigration/minimum wage: .122 (1308)
Great Stagnation/taxes: -.121 (1270)
P(Religion)/P(Warming): -.113 (1505)
P(Supernatural)/taxes: -.113 (1265)
Feminism/Great Stagnation: -.112 (1295)
Number of children/abortion: -.112 (1386)
P(Religion)/basic income: -.108 (1296)
Number of current partners/feminism: .108 (1364)
Basic income/human biodiversity: -.106 (1301)
P(God)/Basic Income: -.105 (1255)
Number of current partners/basic income: .105 (1320)
Human biodiversity/number of languages: .103 (1253)
Number of children/basic income: -.099 (1322)
Number of children/P(Warming): -.091 (1535)

V. Hypothesis Testing

A. Do people in the effective altruism movement donate more money to charity? Do they donate a higher percent of their income to charity? Are they just generally more altruistic people?

1265 people told us how much they give to charity; of those, 450 gave nothing. On average, effective altruists (n = 412) donated $2503 to charity, and other people (n = 853) donated $523  - obviously a significant result. Effective altruists gave on average $800 to MIRI or CFAR, whereas others gave $53. Effective altruists gave on average $16 to other x-risk related charities; others gave only $2.

In order to calculate percent donated I divided charity donations by income in the 947  people helpful enough to give me both numbers. Of those 947, 602 donated nothing to charity, and so had a percent donated of 0. At the other extreme, three  people donated 50% of their (substantial) incomes to charity, and 55 people donated at least 10%. I don't want to draw any conclusions about the community from this because the people who provided both their income numbers and their charity numbers are a highly self-selected sample.

303 effective altruists donated, on average, 3.5% of their income to charity, compared to 645 others who donated, on average, 1% of their income to charity. A small but significant (p < .001) victory for the effective altruism movement.

But are they more compassionate people in general? After throwing out the people who said they wanted to give blood but couldn't for one or another reason, I got 1255 survey respondents giving me an unambiguous answer (yes or no) about whether they'd ever given blood. I found that 51% of effective altruists had given blood compared to 47% of others - a difference which did not reach statistical significance.

Finally, at the end of the survey I had a question offering respondents a chance to cooperate (raising the value of a potential monetary prize to be given out by raffle to a random respondent) or defect (decreasing the value of the prize, but increasing their own chance of winning the raffle). 73% of effective altruists cooperated compared to 70% of others - an insignificant difference.

Conclusion: effective altruists give more money to charity, both absolutely and as a percent of income, but are no more likely (or perhaps only slightly more likely) to be compassionate in other ways.

B. Can we finally resolve this IQ controversy that comes up every year?

The story so far - our first survey in 2009 found an average IQ of 146. Everyone said this was stupid, no community could possibly have that high an average IQ, it was just people lying and/or reporting results from horrible Internet IQ tests.
Although IQ fell somewhat the next few years - to 140 in 2011 and 139 in 2012 - people continued to complain. So in 2012 we started asking for SAT and ACT scores, which are known to correlate well with IQ and are much harder to get wrong. These scores confirmed the 139 IQ result on the 2012 test. But people still objected that something must be up.

This year our IQ has fallen further to 138 (no Flynn Effect for us!) but for the first time we asked people to describe the IQ test they used to get the number. So I took a subset of the people with the most unimpeachable IQ tests - ones taken after the age of 15 (when IQ is more stable), and from a seemingly reputable source. I counted a source as reputable either if it name-dropped a specific scientifically validated IQ test (like WAIS or Raven's Progressive Matrices), if it was performed by a reputable institution (a school, a hospital, or a psychologist), or if it was a Mensa exam proctored by a Mensa official.

This subgroup of 101 people with very reputable IQ tests had an average IQ of 139 - exactly the same as the average among survey respondents as a whole.

I don't know for sure that Mensa is on the level, so I tried again deleting everyone who took a Mensa test - leaving just the people who could name-drop a well-known test or who knew it was administered by a psychologist in an official setting. This caused a precipitous drop all the way down to 138.

The IQ numbers have time and time again answered every challenge raised against them and should be presumed accurate.

C. Can we predict who does or doesn't cooperate on prisoner's dilemmas?

As mentioned above, I included a prisoner's dilemma type question in the survey, offering people the chance to make a little money by screwing all the other survey respondents over.

Tendency to cooperate on the prisoner's dilemma was most highly correlated with items in the general leftist political cluster identified by Ozy above. It was most notable for support for feminism, with which it had a correlation of .15, significant at the p < .01 level, and minimum wage, with which it had a correlation of .09, also significant at p < .01. It was also significantly correlated with belief that other people would cooperate on the same question.

I compared two possible explanations for this result. First, leftists are starry-eyed idealists who believe everyone can just get along - therefore, they expected other people to cooperate more, which made them want to cooperate more. Or, second, most Less Wrongers are white, male, and upper class, meaning that support for leftist values - which often favor nonwhites, women, and the lower class - is itself a symbol of self-sacrifce and altruism which one would expect to correlate with a question testing self-sacrifice and altruism.

I tested the "starry-eyed idealist" hypothesis by checking whether leftists were more likely to believe other people would cooperate. They were not - the correlation was not significant at any level.

I tested the "self-sacrifice" hypothesis by testing whether the feminism correlation went away in women. For women, supporting feminism is presumably not a sign of willingness to self-sacrifice to help an out-group, so we would expect the correlation to disappear.

In the all-female sample, the correlation between feminism and PD cooperation shrunk from .15 to a puny .04, whereas the correlation between the minimum wage and PD was previously .09 and stayed exactly the same at .09. This provides some small level of support for the hypothesis that the leftist correlation with PD cooperation represents a willingness to self-sacrifice in a population who are not themselves helped by leftist values.

(on the other hand, neither leftists nor cooperators were more likely to give money to charity, so if this is true it's a very selective form of self-sacrifice)

VI. Monetary Prize

1389 people answered the prize question at the bottom. 71.6% of these [n = 995] cooperated; 28.4% [n = 394] defected.
The prize goes to a person whose two word phrase begins with "eponymous". If this person posts below (or PMs or emails me) the second word in their phrase, I will give them $60 * 71.6%, or about $43. I can pay to a PayPal account, a charity of their choice that takes online donations, or a snail-mail address via check.

VII. Calibration Questions

The population of Europe, according to designated arbiter Wikipedia, is 739 million people.

People were really really bad at giving their answers in millions. I got numbers anywhere from 3 (really? three million people in Europe?) to 3 billion (3 million billion people = 3 quadrillion). I assume some people thought they were answering in billions, others in thousands, and other people thought they were giving a straight answer in number of individuals.

My original plan was to just adjust these to make them fit, but this quickly encountered some pitfalls. Suppose someone wrote 1 million (as one person did). Could I fairly guess they meant 100 million, even though there's really no way to guess that from the text itself? 1 billion? Maybe they just thought there were really one million people in Europe?

If I was too aggressive correcting these, everyone would get close to the right answer not because they were smart, but because I had corrected their answers. If I wasn't aggressive enough, I would end up with some guy who answered 3 quadrillion Europeans totally distorting the mean.

I ended up deleting 40 answers that suggested there were less than ten million or more than eight billion Europeans, on the grounds that people probably weren't really that far off so it was probably some kind of data entry error, and correcting everyone who entered a reasonable answer in individuals to answer in millions as the question asked.

The remaining 1457 people who can either follow simple directions or at least fail to follow them in a predictable way estimated an average European population in millions of 601 + 35.6 (380, 500, 750).

Respondents were told to aim for within 10% of the real value, which means they wanted between 665 million and 812 million. 18.7% of people [n = 272] got within that window.

I divided people up into calibration brackets of [0,5], [6,15], [16, 25] and so on. The following are what percent of people in each bracket were right.

[0,5]: 7.7%
[6,15]: 12.4%
[16,25]: 15.1%
[26,35]: 18.4%
[36,45]: 20.6%
[46,55]: 15.4%
[56,65]: 16.5%
[66,75]: 21.2%
[76,85]: 36.4%
[86,95]: 48.6%
[96,100]: 100%

Among people who should know better (those who have read all or most of the Sequences and have > 500 karma, a group of 162 people)

[0,5]: 0
[6,15]: 17.4%
[16,25]: 25.6%
[26,35]: 16.7%
[36,45]: 26.7%
[46,55]: 25%
[56,65]: 0%
[66,75]: 8.3%
[76,85]: 40%
[86,95]: 66.6%
[96,100]: 66.6%

Clearly, the people who should know better don't.

This graph represents your performance relative to ideal performance. Dipping below the blue ideal line represents overconfidence; rising above it represents underconfidence. With few exceptions you were very overconfident. Note that there were so few "elite" LWers at certain levels that the graph becomes very noisy and probably isn't representing much; that huge drop at 60 represents like two or three people. The orange "typical LWer" line is much more robust.

There is one other question that gets at the same idea of overconfidence. 651 people were willing to give valid 90% confidence interval on what percent of people would cooperate (this is my fault; I only added this question about halfway through the survey once I realized it would be interesting to investigate). I deleted four for giving extremely high outliers like 9999% which threw off the results, leaving 647 valid answers. The average confidence interval was [28.3, 72.0], which just BARELY contains the correct answer of 71.6%. Of the 647 of you, only 346 (53.5%) gave 90% confidence intervals that included the correct answer!

Last year I complained about horrible performance on calibration questions, but we all decided it was probably just a fluke caused by a particularly weird question. This year's results suggest that was no fluke and that we haven't even learned to overcome the one bias that we can measure super-well and which is most easily trained away. Disappointment!

VIII. Public Data

There's still a lot more to be done with this survey. User:Unnamed has promised to analyze the "Extra Credit: CFAR Questions" section (not included in this post), but so far no one has looked at the "Extra Credit: Questions From Sarah" section, which I didn't really know what to do with. And of course this is most complete survey yet for seeking classic findings like "People who disagree with me about politics are stupid and evil".

1480 people - over 90% of the total - kindly allowed me to make their survey data public. I have included all their information except the timestamp (which would make tracking pretty easy) including their secret passphrases (by far the most interesting part of this exercise was seeing what unusual two word phrases people could come up with on short notice).

Comment author: Vaniver 19 January 2014 09:16:43PM 5 points [-]

Actually, I just ran the numbers on the SAT2400 and they're closer; the average percentile predicted from that is 99th, which corresponds to about 135.

Comment author: Yvain 20 January 2014 02:41:20AM 1 point [-]

One reason SAT1600 and SAT2400 scores may differ is that some of the SAT1600 scores might in fact have come from before the 1994 renorming. Have you tried doing pre-1994 and post-1994 scores separately (guessing when someone took the SAT based on age?)

In response to comment by RRand on 2013 Survey Results
Comment author: jkaufman 19 January 2014 03:52:10PM 0 points [-]

How is it that 100% of the general population with high (>96%) confidence got the correct answer, but only 66% of a subset of that population?

What if it was divided into (typical-lw) (elite-lw) not (typical-lw (elite-lw))? That is, disjoint sets not subsets.

Comment author: Yvain 19 January 2014 04:08:09PM 0 points [-]

I think it's more likely that I accidentally did 95-100 inclusive for one and 95-100 exclusive for the other.

2013 Survey Results

74 Post author: Yvain 19 January 2014 02:51AM

Thanks to everyone who took the 2013 Less Wrong Census/Survey. Extra thanks to Ozy, who helped me out with the data processing and statistics work, and to everyone who suggested questions.

This year's results are below. Some of them may make more sense in the context of the original survey questions, which can be seen here. Please do not try to take the survey as it is over and your results will not be counted.

Part I. Population

1636 people answered the survey.

Compare this to 1195 people last year, and 1090 people the year before that. It would seem the site is growing, but we do have to consider that each survey lasted a different amount of time; for example, last survey lasted 23 days, but this survey lasted 40.

However, almost everyone who takes the survey takes it in the first few weeks it is available. 1506 of the respondents answered within the first 23 days, proving that even if the survey ran the same length as last year's, there would still have been growth.
As we will see lower down, growth is smooth across all categories of users (lurkers, commenters, posters) EXCEPT people who have posted to Main, the number of which remains nearly the same from year to year.

We continue to have very high turnover - only 40% of respondents this year say they also took the survey last year.

II. Categorical Data

SEX:
Female: 161, 9.8%
Male: 1453, 88.8%
Other: 1, 0.1%
Did not answer: 21, 1.3%

[[Ozy is disappointed that we've lost 50% of our intersex readers.]]

GENDER:
F (cisgender): 140, 8.6%
F (transgender MtF): 20, 1.2%
M (cisgender): 1401, 85.6%
M (transgender FtM): 5, 0.3%
Other: 49, 3%
Did not answer: 21, 1.3%

SEXUAL ORIENTATION:
Asexual: 47, 2.9%
Bisexual: 188, 12.2%
Heterosexual: 1287, 78.7%
Homosexual: 45, 2.8%
Other: 39, 2.4%
Did not answer: 19, 1.2%

RELATIONSHIP STYLE:
Prefer monogamous: 829, 50.7%
Prefer polyamorous: 234, 14.3%
Other: 32, 2.0%
Uncertain/no preference: 520, 31.8%
Did not answer: 21, 1.3%

NUMBER OF CURRENT PARTNERS:
0: 797, 48.7%
1: 728, 44.5%
2: 66, 4.0%
3: 21, 1.3%
4: 1, .1%
6: 3, .2%
Did not answer: 20, 1.2%

RELATIONSHIP STATUS:
Married: 304, 18.6%
Relationship: 473, 28.9%
Single: 840, 51.3%

RELATIONSHIP GOALS:
Looking for more relationship partners: 617, 37.7%
Not looking for more relationship partners: 993, 60.7%
Did not answer: 26, 1.6%

HAVE YOU DATED SOMEONE YOU MET THROUGH THE LESS WRONG COMMUNITY?
Yes: 53, 3.3%
I didn't meet them through the community but they're part of the community now: 66, 4.0%
No: 1482, 90.5%
Did not answer: 35, 2.1%

COUNTRY:
United States: 895, 54.7%
United Kingdom: 144, 8.8%
Canada: 107, 6.5%
Australia: 69, 4.2%
Germany: 68, 4.2%
Finland: 35, 2.1%
Russia: 22, 1.3%
New Zealand: 20, 1.2%
Israel: 17, 1.0%
France: 16, 1.0%
Poland: 16, 1.0%

LESS WRONGERS PER CAPITA:
Finland: 1/154,685.
New Zealand: 1/221,650.
Canada: 1/325,981.
Australia: 1/328,659.
United States: 1/350,726
United Kingdom: 1/439,097
Israel: 1/465,176.
Germany: 1/1,204,264.
Poland: 1/2,408,750.
France: 1/4,106,250.
Russia: 1/6,522,727

RACE:
Asian (East Asian): 60, 3.7%
Asian (Indian subcontinent): 37, 2.3%
Black: 11, .7%
Middle Eastern: 9, .6%
White (Hispanic): 73, 4.5%
White (non-Hispanic): 1373, 83.9%
Other: 51, 3.1%
Did not answer: 22, 1.3%

WORK STATUS:
Academics (teaching): 77, 4.7%
For-profit work: 552, 33.7%
Government work: 55, 3.4%
Independently wealthy: 14, .9%
Non-profit work: 46, 2.8%
Self-employed: 103, 6.3%
Student: 661, 40.4%
Unemployed: 105, 6.4%
Did not answer: 23, 1.4%

PROFESSION:
Art: 27, 1.7%
Biology: 26, 1.6%
Business: 44, 2.7%
Computers (AI): 47, 2.9%
Computers (other academic computer science): 107, 6.5%
Computers (practical): 505, 30.9%
Engineering: 128, 7.8%
Finance/economics: 92, 5.6%
Law: 36, 2.2%
Mathematics: 139, 8.5%
Medicine: 31, 1.9%
Neuroscience: 13, .8%
Philosophy: 41, 2.5%
Physics: 92, 5.6%
Psychology: 34, 2.1%
Statistics: 23, 1.4%
Other hard science: 31, 1.9%
Other social science: 43, 2.6%
Other: 139, 8.5%
Did not answer: 38, 2.3%

DEGREE:
None: 84, 5.1%
High school: 444, 27.1%
2 year degree: 68, 4.2%
Bachelor's: 554, 33.9%
Master's: 323, 19.7%
MD/JD/other professional degree: 31, 2.0%
PhD.: 90, 5.5%
Other: 22, 1.3%
Did not answer: 19, 1.2%

POLITICAL:
Communist: 11, .7%
Conservative: 64, 3.9%
Liberal: 580, 35.5%
Libertarian: 437, 26.7%
Socialist: 502, 30.7%
Did not answer: 42, 2.6%

COMPLEX POLITICAL WITH WRITE-IN:
Anarchist: 52, 3.2%
Conservative: 16, 1.0%
Futarchist: 42, 2.6%
Left-libertarian: 142, 8.7%
Liberal: 5
Moderate: 53, 3.2%
Pragmatist: 110, 6.7%
Progressive: 206, 12.6%
Reactionary: 40, 2.4%
Social democrat: 154, 9.5%
Socialist: 135, 8.2%
Did not answer: 26.2%

[[All answers with more than 1% of the Less Wrong population included. Other answers which made Ozy giggle included "are any of you kings?! why do you CARE?!", "Exclusionary: you are entitled to an opinion on nuclear power when you know how much of your power is nuclear", "having-well-founded-opinions-is-really-hard-ist", "kleptocrat", "pirate", and "SPECIAL FUCKING SNOWFLAKE."]]

AMERICAN PARTY AFFILIATION:
Democratic Party: 226, 13.8%
Libertarian Party: 31, 1.9%
Republican Party: 58, 3.5%
Other third party: 19, 1.2%
Not registered: 447, 27.3%
Did not answer or non-American: 856, 52.3%

VOTING:
Yes: 936, 57.2%
No: 450, 27.5%
My country doesn't hold elections: 2, 0.1%
Did not answer: 249, 15.2%

RELIGIOUS VIEWS:
Agnostic: 165, 10.1%
Atheist and not spiritual: 1163, 71.1%
Atheist but spiritual: 132, 8.1%
Deist/pantheist/etc.: 36, 2.2%
Lukewarm theist: 53, 3.2%
Committed theist 64, 3.9%

RELIGIOUS DENOMINATION (IF THEIST):
Buddhist: 22, 1.3%
Christian (Catholic): 44, 2.7%
Christian (Protestant): 56, 3.4%
Jewish: 31, 1.9%
Mixed/Other: 21, 1.3%
Unitarian Universalist or similar: 25, 1.5%

[[This includes all religions with more than 1% of Less Wrongers. Minority religions include Dzogchen, Daoism, various sorts of Paganism, Simulationist, a very confused secular humanist, Kopmist, Discordian, and a Cultus Deorum Romanum practitioner whom Ozy wants to be friends with.]]

FAMILY RELIGION:
Agnostic: 129, 11.6%
Atheist and not spiritual: 225, 13.8%
Atheist but spiritual: 73, 4.5%
Committed theist: 423, 25.9%
Deist/pantheist, etc.: 42, 2.6%
Lukewarm theist: 563, 34.4%
Mixed/other: 97, 5.9%
Did not answer: 24, 1.5%

RELIGIOUS BACKGROUND:
Bahai: 3, 0.2%
Buddhist: 13, .8%
Christian (Catholic): 418, 25.6%
Christian (Mormon): 38, 2.3%
Christian (Protestant): 631, 38.4%
Christian (Quaker): 7, 0.4%
Christian (Unitarian Universalist or similar): 32, 2.0%
Christian (other non-Protestant): 99, 6.1%
Christian (unknown): 3, 0.2%
Eckankar: 1, 0.1%
Hindu: 29, 1.8%
Jewish: 136, 8.3%
Muslim: 12, 0.7%
Native American Spiritualist: 1, 0.1%
Mixed/Other: 85, 5.3%
Sikhism: 1, 0.1%
Traditional Chinese: 11, .7%
Wiccan: 1, 0.1%
None: 8, 0.4%
Did not answer: 107, 6.7%

MORAL VIEWS:
Accept/lean towards consequentialism: 1049, 64.1%
Accept/lean towards deontology: 77, 4.7%
Accept/lean towards virtue ethics: 197, 12.0%
Other/no answer: 276, 16.9%
Did not answer: 37, 2.3%

CHILDREN
0: 1414, 86.4%
1: 77, 4.7%
2: 90, 5.5%
3: 25, 1.5%
4: 7, 0.4%
5: 1, 0.1%
6: 2, 0.1%
Did not answer: 20, 1.2%

MORE CHILDREN:
Have no children, don't want any: 506, 31.3%
Have no children, uncertain if want them: 472, 29.2%
Have no children, want children: 431, 26.7%
Have no children, didn't answer: 5, 0.3%
Have children, don't want more: 124, 7.6%
Have children, uncertain if want more: 25, 1.5%
Have children, want more: 53, 3.2%

HANDEDNESS:
Right: 1256, 76.6%
Left: 145, 9.5%
Ambidextrous: 36, 2.2%
Not sure: 7, 0.4%
Did not answer: 182, 11.1%

LESS WRONG USE:
Lurker (no account): 584, 35.7%
Lurker (account) 221, 13.5%
Poster (comment, no post): 495, 30.3%
Poster (Discussion, not Main): 221, 12.9%
Poster (Main): 103, 6.3%

SEQUENCES:
Never knew they existed: 119, 7.3%
Knew they existed, didn't look at them: 48, 2.9%
~25% of the Sequences: 200, 12.2%
~50% of the Sequences: 271, 16.6%
~75% of the Sequences: 225, 13.8%
All the Sequences: 419, 25.6%
Did not answer: 24, 1.5%

MEETUPS:
No: 1134, 69.3%
Yes, once or a few times: 307, 18.8%
Yes, regularly: 159, 9.7%

HPMOR:
No: 272, 16.6%
Started it, haven't finished: 255, 15.6%
Yes, all of it: 912, 55.7%

CFAR WORKSHOP ATTENDANCE:
Yes, a full workshop: 105, 6.4%
A class but not a full-day workshop: 40, 2.4%
No: 1446, 88.3%
Did not answer: 46, 2.8%

PHYSICAL INTERACTION WITH LW COMMUNITY:
Yes, all the time: 94, 5.7%
Yes, sometimes: 179, 10.9%
No: 1316, 80.4%
Did not answer: 48, 2.9%

VEGETARIAN:
No: 1201, 73.4%
Yes: 213, 13.0%
Did not answer: 223, 13.6%

SPACED REPETITION:
Never heard of them: 363, 22.2%
No,  but I've heard of them: 495, 30.2%
Yes, in the past: 328, 20%
Yes, currently: 219, 13.4%
Did not answer: 232, 14.2%

HAVE YOU TAKEN PREVIOUS INCARNATIONS OF THE LESS WRONG SURVEY?
Yes: 638, 39.0%
No: 784, 47.9%
Did not answer: 215, 13.1%

PRIMARY LANGUAGE:
English: 1009, 67.8%
German: 58, 3.6%
Finnish: 29, 1.8%
Russian: 25, 1.6%
French: 17, 1.0%
Dutch: 16, 1.0%
Did not answer: 15.2%

[[This includes all answers that more than 1% of respondents chose. Other languages include Urdu, both Czech and Slovakian, Latvian, and Love.]]

ENTREPRENEUR:
I don't want to start my own business: 617, 37.7%
I am considering starting my own business: 474, 29.0%
I plan to start my own business: 113, 6.9%
I've already started my own business: 156, 9.5%
Did not answer: 277, 16.9%

EFFECTIVE ALTRUIST:
Yes: 468, 28.6%
No: 883, 53.9%
Did not answer: 286, 17.5%

WHO ARE YOU LIVING WITH?
Alone: 348, 21.3%
With family: 420, 25.7%
With partner/spouse: 400, 24.4%
With roommates: 450, 27.5%
Did not answer: 19, 1.3%

DO YOU GIVE BLOOD?
No: 646, 39.5%
No, only because I'm not allowed: 157, 9.6%
Yes, 609, 37.2%
Did not answer: 225, 13.7%

GLOBAL CATASTROPHIC RISK:
Pandemic (bioengineered): 374, 22.8%
Environmental collapse including global warming: 251, 15.3%
Unfriendly AI: 233, 14.2%
Nuclear war: 210, 12.8%
Pandemic (natural) 145, 8.8%
Economic/political collapse: 175, 1, 10.7%
Asteroid strike: 65, 3.9%
Nanotech/grey goo: 57, 3.5%
Didn't answer: 99, 6.0%

CRYONICS STATUS:
Never thought about it / don't understand it: 69, 4.2%
No, and don't want to: 414, 25.3%
No, still considering: 636, 38.9%
No, would like to: 265, 16.2%
No, would like to, but it's unavailable: 119, 7.3%
Yes: 66, 4.0%
Didn't answer: 68, 4.2%

NEWCOMB'S PROBLEM:
Don't understand/prefer not to answer: 92, 5.6%
Not sure: 103, 6.3%
One box: 1036, 63.3%
Two box: 119, 7.3%
Did not answer: 287, 17.5%

GENOMICS:
Yes: 177, 10.8%
No: 1219, 74.5%
Did not answer: 241, 14.7%

REFERRAL TYPE:
Been here since it started in the Overcoming Bias days: 285, 17.4%
Referred by a friend: 241, 14.7%
Referred by a search engine: 148, 9.0%
Referred by HPMOR: 400, 24.4%
Referred by a link on another blog: 373, 22.8%
Referred by a school course: 1, .1%
Other: 160, 9.8%
Did not answer: 29, 1.9%

REFERRAL SOURCE:
Common Sense Atheism: 33
Slate Star Codex: 20
Hacker News: 18
Reddit: 18
TVTropes: 13
Y Combinator: 11
Gwern: 9
RationalWiki: 8
Marginal Revolution: 7
Unequally Yoked: 6
Armed and Dangerous: 5
Shtetl Optimized: 5
Econlog: 4
StumbleUpon: 4
Yudkowsky.net: 4
Accelerating Future: 3
Stares at the World: 3
xkcd: 3
David Brin: 2
Freethoughtblogs: 2
Felicifia: 2
Givewell: 2
hatrack.com: 2
HPMOR: 2
Patri Friedman: 2
Popehat: 2
Overcoming Bias: 2
Scientiststhesis: 2
Scott Young: 2
Stardestroyer.net: 2
TalkOrigins: 2
Tumblr: 2

[[This includes all sources with  more than one referral; needless to say there was a long tail]]

III. Numeric Data

(in the form mean + stdev (1st quartile, 2nd quartile, 3rd quartile) [n = number responding]))

Age: 27.4 + 8.5 (22, 25, 31) [n = 1558]
Height: 176.6 cm + 16.6 (173, 178, 183) [n = 1267]

Karma Score: 504 + 2085 (0, 0, 100) [n = 1438]
Time in community: 2.62 years + 1.84 (1, 2, 4) [n = 1443]
Time on LW: 13.25 minutes/day + 20.97 (2, 10, 15) [n = 1457]

IQ: 138.2 + 13.6 (130, 138, 145) [n = 506]
SAT out of 1600: 1474 + 114 (1410, 1490, 1560) [n = 411]
SAT out of 2400: 2207 + 161 (2130, 2240, 2330) [n = 333]
ACT out of 36: 32.8 + 2.5 (32, 33, 35) [n = 265]

P(Aliens in observable universe): 74.3 + 32.7 (60, 90, 99) [n = 1496]
P(Aliens in Milky Way): 44.9 + 38.2 (5, 40, 85) [n = 1482]
P(Supernatural): 7.7 + 22 (0E-9, .000055, 1) [n = 1484]
P(God): 9.1 + 22.9 (0E-11, .01, 3) [n = 1490]
P(Religion): 5.6 + 19.6 (0E-11, 0E-11, .5) [n = 1497]
P(Cryonics): 22.8 + 28 (2, 10, 33) [n = 1500]  
P(AntiAgathics): 27.6 + 31.2 (2, 10, 50) [n = 1493]
P(Simulation): 24.1 + 28.9 (1, 10, 50) [n = 1400]
P(ManyWorlds): 50 + 29.8 (25, 50, 75) [n = 1373]
P(Warming): 80.7 + 25.2 (75, 90, 98) [n = 1509]
P(Global catastrophic risk): 72.9 + 25.41 (60, 80, 95) [n = 1502]
Singularity year: 1.67E +11 + 4.089E+12 (2060, 2090, 2150) [n = 1195]

[[Of course, this question was hopelessly screwed up by people who insisted on filling the whole answer field with 9s, or other such nonsense. I went back and eliminated all outliers - answers with more than 4 digits or answers in the past - which changed the results to: 2150 + 226 (2060, 2089, 2150)]]

Yearly Income: $73,226 +423,310 (10,000, 37,000, 80,000) [n = 910]
Yearly Charity: $1181.16 + 6037.77 (0, 50, 400) [n = 1231]
Yearly Charity to MIRI/CFAR: $307.18 + 4205.37 (0, 0, 0) [n = 1191]
Yearly Charity to X-risk (excluding MIRI or CFAR): $6.34 + 55.89 (0, 0, 0) [n = 1150]

Number of Languages: 1.49 + .8 (1, 1, 2) [n = 1345]
Older Siblings: 0.5 + 0.9 (0, 0, 1) [n = 1366]
Time Online/Week: 42.7 hours + 24.8 (25, 40, 60) [n = 1292]
Time Watching TV/Week: 4.2 hours + 5.7 (0, 2, 5) [n = 1316]

[[The next nine questions ask respondents to rate how favorable they are to the political idea or movement above on a scale of 1 to 5, with 1 being "not at all favorable" and 5 being "very favorable". You can see the exact wordings of the questions on the survey.]]

Abortion: 4.4 + 1 (4, 5, 5) [n = 1350]
Immigration: 4.1 + 1 (3, 4, 5) [n = 1322]
Basic Income: 3.8 + 1.2 (3, 4, 5) [n = 1289]
Taxes: 3.1 + 1.3 (2, 3, 4) [n = 1296]
Feminism: 3.8 + 1.2 (3, 4, 5) [n = 1329]
Social Justice: 3.6 + 1.3 (3, 4, 5) [n = 1263]
Minimum Wage: 3.2 + 1.4 (2, 3, 4) [n = 1290]
Great Stagnation: 2.3 + 1 (2, 2, 3) [n = 1273]
Human Biodiversity: 2.7 + 1.2 (2, 3, 4) [n = 1305]

IV. Bivariate Correlations

Ozy ran bivariate correlations between all the numerical data and recorded all correlations that were significant at the .001 level in order to maximize the chance that these are genuine results. The format is variable/variable: Pearson correlation (n). Yvain is not hugely on board with the idea of running correlations between everything and seeing what sticks, but will grudgingly publish the results because of the very high bar for significance (p < .001 on ~800 correlations suggests < 1 spurious result) and because he doesn't want to have to do it himself.

Less Political:
SAT score (1600)/SAT score (2400): .835 (56)
Charity/MIRI and CFAR donations: .730 (1193)
SAT score out of 2400/ACT score: .673 (111)
SAT score out of 1600/ACT score: .544 (102)
Number of children/age: .507 (1607)
P(Cryonics)/P(AntiAgathics): .489 (1515)
SAT score out of 1600/IQ: .369 (173)
MIRI and CFAR donations/XRisk donations: .284 (1178)
Number of children/ACT score: -.279 (269)
Income/charity: .269 (884)
Charity/Xrisk charity: .262 (1161)
P(Cryonics)/P(Simulation): .256 (1419)
P(AntiAgathics)/P(Simulation): .253 (1418)
Number of current partners/age: .238 (1607) 
Number of children/SAT score (2400): -.223 (345)
Number of current partners/number of children: .205 (1612)
SAT score out of 1600/age: -.194 (422)
Charity/age: .175 (1259)
Time on Less Wrong/IQ: -.164 (492)
P(Warming)/P(GlobalCatastrophicRisk): .156 (1522)
Number of current partners/IQ: .155 (521)
P(Simulation)/age: -.153 (1420)
Immigration/P(ManyWorlds): .150 (1195)
Income/age: .150 (930)
P(Cryonics)/age: -.148 (1521)
Income/children: .145 (931)
P(God)/P(Simulation): .142 (1409)
Number of children/P(Aliens): .140 (1523)
P(AntiAgathics)/Hours Online: .138 (1277)
Number of current partners/karma score: .137 (1470)
Abortion/P(ManyWorlds): .122 (1215)
Feminism/Xrisk charity donations: -.122 (1104)
P(AntiAgathics)/P(ManyWorlds) .118 (1381)
P(Cryonics)/P(ManyWorlds): .117 (1387)
Karma score/Great Stagnation: .114 (1202)
Hours online/P(simulation): .114 (1199)
P(Cryonics)/Hours Online: .113 (1279)
P(AntiAgathics)/Great Stagnation: -.111 (1259)
Basic income/hours online: .111 (1200)
P(GlobalCatastrophicRisk)/Great Stagnation: -.110 (1270)
Age/X risk charity donations: .109 (1176)
P(AntiAgathics)/P(GlobalCatastrophicRisk): -.109 (1513)
Time on Less Wrong/age: -.108 (1491)
P(AntiAgathics)/Human Biodiversity: .104 (1286)
Immigration/Hours Online: .104 (1226)
P(Simulation)/P(GlobalCatastrophicRisk): -.103 (1421)
P(Supernatural)/height: -.101 (1232)
P(GlobalCatastrophicRisk)/height: .101 (1249)
Number of children/hours online: -.099 (1321)
P(AntiAgathics)/age: -.097 (1514)
Karma score/time on LW: .096 (1404)

This year for the first time P(Aliens) and P(Aliens2) are entirely uncorrelated with each other. Time in Community, Time on LW, and IQ are not correlated with anything particularly interesting, suggesting all three fail to change people's views.

Results we find amusing: high-IQ and high-karma people have more romantic partners, suggesting that those are attractive traits. There is definitely a Cryonics/Antiagathics/Simulation/Many Worlds cluster of weird beliefs, which younger people and people who spend more time online are slightly more likely to have - weirdly, that cluster seems slightly less likely to believe in global catastrophic risk. Older people and people with more children have more romantic partners (it'd be interesting to see if that holds true for the polyamorous). People who believe in anti-agathics and global catastrophic risk are less likely to believe in a great stagnation (presumably because both of the above rely on inventions). People who spend more time on Less Wrong have lower IQs. Height is, bizarrely, correlated with belief in the supernatural and global catastrophic risk.

All political viewpoints are correlated with each other in pretty much exactly the way one would expect. They are also correlated with one's level of belief in God, the supernatural, and religion. There are minor correlations with some of the beliefs and number of partners (presumably because polyamory), number of children, and number of languages spoken. We are doing terribly at avoiding Blue/Green politics, people.

More Political:
P(Supernatural)/P(God): .736 (1496)
P(Supernatural)/P(Religion): .667 (1492)
Minimum wage/taxes: .649 (1299)
P(God)/P(Religion): .631 (1496)
Feminism/social justice: .619 (1293)
Social justice/minimum wage: .508 (1262)
P(Supernatural)/abortion: -.469 (1309)
Taxes/basic income: .463 (1285)
P(God)/abortion: -.461 (1310)
Social justice/taxes: .456 (1267)
P(Religion)/abortion: -.413
Basic income/minimum wage: .392 (1283)
Feminism/taxes: .391 (1318)
Feminism/minimum wage: .391 (1312)
Feminism/human biodiversity: -.365 (1331)
Immigration/feminism: .355 (1336)
P(Warming)/taxes: .340 (1292)
Basic income/social justice: .311 (1270)
Immigration/social justice: .307 (1275)
P(Warming)/feminism: .294 (1323)
Immigration/human biodiversity: -.292 (1313)
P(Warming)/basic income: .290 (1287)
Social justice/human biodiversity: -.289 (1281)
Basic income/feminism: .284 (1313)
Human biodiversity/minimum wage: -.273 (1293)
P(Warming)/social justice: .271 (1261)
P(Warming)/minimum wage: .262 (1284)
Human biodiversity/taxes: -.251 (1270).
Abortion/feminism: .239 (1356)
Abortion/social justice: .220 (1292)
P(Warming)/immigration: .215 (1315)
Abortion/immigration: .211 (1353)
P(Warming)/abortion: .192 (1340)
Immigration/taxes: .186 (1322)
Basic income/taxes: .174 (1249)
Abortion/taxes: .170 (1328)
Abortion/minimum wage: .169 (1317)
P(warming)/human biodiversity: -.168 (1301)
Abortion/basic income: .168 (1314)
Immigration/Great Stagnation: -.163 (1281)
P(God)/feminism: -.159 (1294)
P(Supernatural)/feminism: -.158 (1292)
Human biodiversity/Great Stagnation: .152 (1287)
Social justice/Great Stagnation: -.135 (1242)
Number of languages/taxes: -.133 (1242)
P(God)/P(Warming): -.132 (1491)
P(Supernatural)/immigration: -.131 (1284)
P(Religion)immigration: -.129 (1296)
P(God)/immigration: -.127 (1286)
P(Supernatural)/P(Warming): -.125 (1487)
P(Supernatural)/social justice: -.125 (1227)
P(God)/taxes: -.145
Minimum wage/Great Stagnation: -124 (1269)
Immigration/minimum wage: .122 (1308)
Great Stagnation/taxes: -.121 (1270)
P(Religion)/P(Warming): -.113 (1505)
P(Supernatural)/taxes: -.113 (1265)
Feminism/Great Stagnation: -.112 (1295)
Number of children/abortion: -.112 (1386)
P(Religion)/basic income: -.108 (1296)
Number of current partners/feminism: .108 (1364)
Basic income/human biodiversity: -.106 (1301)
P(God)/Basic Income: -.105 (1255)
Number of current partners/basic income: .105 (1320)
Human biodiversity/number of languages: .103 (1253)
Number of children/basic income: -.099 (1322)
Number of children/P(Warming): -.091 (1535)

V. Hypothesis Testing

A. Do people in the effective altruism movement donate more money to charity? Do they donate a higher percent of their income to charity? Are they just generally more altruistic people?

1265 people told us how much they give to charity; of those, 450 gave nothing. On average, effective altruists (n = 412) donated $2503 to charity, and other people (n = 853) donated $523  - obviously a significant result. Effective altruists gave on average $800 to MIRI or CFAR, whereas others gave $53. Effective altruists gave on average $16 to other x-risk related charities; others gave only $2.

In order to calculate percent donated I divided charity donations by income in the 947  people helpful enough to give me both numbers. Of those 947, 602 donated nothing to charity, and so had a percent donated of 0. At the other extreme, three  people donated 50% of their (substantial) incomes to charity, and 55 people donated at least 10%. I don't want to draw any conclusions about the community from this because the people who provided both their income numbers and their charity numbers are a highly self-selected sample.

303 effective altruists donated, on average, 3.5% of their income to charity, compared to 645 others who donated, on average, 1% of their income to charity. A small but significant (p < .001) victory for the effective altruism movement.

But are they more compassionate people in general? After throwing out the people who said they wanted to give blood but couldn't for one or another reason, I got 1255 survey respondents giving me an unambiguous answer (yes or no) about whether they'd ever given blood. I found that 51% of effective altruists had given blood compared to 47% of others - a difference which did not reach statistical significance.

Finally, at the end of the survey I had a question offering respondents a chance to cooperate (raising the value of a potential monetary prize to be given out by raffle to a random respondent) or defect (decreasing the value of the prize, but increasing their own chance of winning the raffle). 73% of effective altruists cooperated compared to 70% of others - an insignificant difference.

Conclusion: effective altruists give more money to charity, both absolutely and as a percent of income, but are no more likely (or perhaps only slightly more likely) to be compassionate in other ways.

B. Can we finally resolve this IQ controversy that comes up every year?

The story so far - our first survey in 2009 found an average IQ of 146. Everyone said this was stupid, no community could possibly have that high an average IQ, it was just people lying and/or reporting results from horrible Internet IQ tests.
Although IQ fell somewhat the next few years - to 140 in 2011 and 139 in 2012 - people continued to complain. So in 2012 we started asking for SAT and ACT scores, which are known to correlate well with IQ and are much harder to get wrong. These scores confirmed the 139 IQ result on the 2012 test. But people still objected that something must be up.

This year our IQ has fallen further to 138 (no Flynn Effect for us!) but for the first time we asked people to describe the IQ test they used to get the number. So I took a subset of the people with the most unimpeachable IQ tests - ones taken after the age of 15 (when IQ is more stable), and from a seemingly reputable source. I counted a source as reputable either if it name-dropped a specific scientifically validated IQ test (like WAIS or Raven's Progressive Matrices), if it was performed by a reputable institution (a school, a hospital, or a psychologist), or if it was a Mensa exam proctored by a Mensa official.

This subgroup of 101 people with very reputable IQ tests had an average IQ of 139 - exactly the same as the average among survey respondents as a whole.

I don't know for sure that Mensa is on the level, so I tried again deleting everyone who took a Mensa test - leaving just the people who could name-drop a well-known test or who knew it was administered by a psychologist in an official setting. This caused a precipitous drop all the way down to 138.

The IQ numbers have time and time again answered every challenge raised against them and should be presumed accurate.

C. Can we predict who does or doesn't cooperate on prisoner's dilemmas?

As mentioned above, I included a prisoner's dilemma type question in the survey, offering people the chance to make a little money by screwing all the other survey respondents over.

Tendency to cooperate on the prisoner's dilemma was most highly correlated with items in the general leftist political cluster identified by Ozy above. It was most notable for support for feminism, with which it had a correlation of .15, significant at the p < .01 level, and minimum wage, with which it had a correlation of .09, also significant at p < .01. It was also significantly correlated with belief that other people would cooperate on the same question.

I compared two possible explanations for this result. First, leftists are starry-eyed idealists who believe everyone can just get along - therefore, they expected other people to cooperate more, which made them want to cooperate more. Or, second, most Less Wrongers are white, male, and upper class, meaning that support for leftist values - which often favor nonwhites, women, and the lower class - is itself a symbol of self-sacrifce and altruism which one would expect to correlate with a question testing self-sacrifice and altruism.

I tested the "starry-eyed idealist" hypothesis by checking whether leftists were more likely to believe other people would cooperate. They were not - the correlation was not significant at any level.

I tested the "self-sacrifice" hypothesis by testing whether the feminism correlation went away in women. For women, supporting feminism is presumably not a sign of willingness to self-sacrifice to help an out-group, so we would expect the correlation to disappear.

In the all-female sample, the correlation between feminism and PD cooperation shrunk from .15 to a puny .04, whereas the correlation between the minimum wage and PD was previously .09 and stayed exactly the same at .09. This provides some small level of support for the hypothesis that the leftist correlation with PD cooperation represents a willingness to self-sacrifice in a population who are not themselves helped by leftist values.

(on the other hand, neither leftists nor cooperators were more likely to give money to charity, so if this is true it's a very selective form of self-sacrifice)

VI. Monetary Prize

1389 people answered the prize question at the bottom. 71.6% of these [n = 995] cooperated; 28.4% [n = 394] defected.
The prize goes to a person whose two word phrase begins with "eponymous". If this person posts below (or PMs or emails me) the second word in their phrase, I will give them $60 * 71.6%, or about $43. I can pay to a PayPal account, a charity of their choice that takes online donations, or a snail-mail address via check.

VII. Calibration Questions

The population of Europe, according to designated arbiter Wikipedia, is 739 million people.

People were really really bad at giving their answers in millions. I got numbers anywhere from 3 (really? three million people in Europe?) to 3 billion (3 million billion people = 3 quadrillion). I assume some people thought they were answering in billions, others in thousands, and other people thought they were giving a straight answer in number of individuals.

My original plan was to just adjust these to make them fit, but this quickly encountered some pitfalls. Suppose someone wrote 1 million (as one person did). Could I fairly guess they meant 100 million, even though there's really no way to guess that from the text itself? 1 billion? Maybe they just thought there were really one million people in Europe?

If I was too aggressive correcting these, everyone would get close to the right answer not because they were smart, but because I had corrected their answers. If I wasn't aggressive enough, I would end up with some guy who answered 3 quadrillion Europeans totally distorting the mean.

I ended up deleting 40 answers that suggested there were less than ten million or more than eight billion Europeans, on the grounds that people probably weren't really that far off so it was probably some kind of data entry error, and correcting everyone who entered a reasonable answer in individuals to answer in millions as the question asked.

The remaining 1457 people who can either follow simple directions or at least fail to follow them in a predictable way estimated an average European population in millions of 601 + 35.6 (380, 500, 750).

Respondents were told to aim for within 10% of the real value, which means they wanted between 665 million and 812 million. 18.7% of people [n = 272] got within that window.

I divided people up into calibration brackets of [0,5], [6,15], [16, 25] and so on. The following are what percent of people in each bracket were right.

[0,5]: 7.7%
[6,15]: 12.4%
[16,25]: 15.1%
[26,35]: 18.4%
[36,45]: 20.6%
[46,55]: 15.4%
[56,65]: 16.5%
[66,75]: 21.2%
[76,85]: 36.4%
[86,95]: 48.6%
[96,100]: 100%

Among people who should know better (those who have read all or most of the Sequences and have > 500 karma, a group of 162 people)

[0,5]: 0
[6,15]: 17.4%
[16,25]: 25.6%
[26,35]: 16.7%
[36,45]: 26.7%
[46,55]: 25%
[56,65]: 0%
[66,75]: 8.3%
[76,85]: 40%
[86,95]: 66.6%
[96,100]: 66.6%

Clearly, the people who should know better don't.

This graph represents your performance relative to ideal performance. Dipping below the blue ideal line represents overconfidence; rising above it represents underconfidence. With few exceptions you were very overconfident. Note that there were so few "elite" LWers at certain levels that the graph becomes very noisy and probably isn't representing much; that huge drop at 60 represents like two or three people. The orange "typical LWer" line is much more robust.

There is one other question that gets at the same idea of overconfidence. 651 people were willing to give valid 90% confidence interval on what percent of people would cooperate (this is my fault; I only added this question about halfway through the survey once I realized it would be interesting to investigate). I deleted four for giving extremely high outliers like 9999% which threw off the results, leaving 647 valid answers. The average confidence interval was [28.3, 72.0], which just BARELY contains the correct answer of 71.6%. Of the 647 of you, only 346 (53.5%) gave 90% confidence intervals that included the correct answer!

Last year I complained about horrible performance on calibration questions, but we all decided it was probably just a fluke caused by a particularly weird question. This year's results suggest that was no fluke and that we haven't even learned to overcome the one bias that we can measure super-well and which is most easily trained away. Disappointment!

VIII. Public Data

There's still a lot more to be done with this survey. User:Unnamed has promised to analyze the "Extra Credit: CFAR Questions" section (not included in this post), but so far no one has looked at the "Extra Credit: Questions From Sarah" section, which I didn't really know what to do with. And of course this is most complete survey yet for seeking classic findings like "People who disagree with me about politics are stupid and evil".

1480 people - over 90% of the total - kindly allowed me to make their survey data public. I have included all their information except the timestamp (which would make tracking pretty easy) including their secret passphrases (by far the most interesting part of this exercise was seeing what unusual two word phrases people could come up with on short notice).

Comment author: Alexei 09 January 2014 12:20:23AM 2 points [-]

Wow, this is exactly opposite of what I expected. Thank you!

Comment author: Yvain 09 January 2014 12:26:45AM 5 points [-]

You expected men to be more feminist than women? Why?

Comment author: Alexei 08 January 2014 06:58:54PM 3 points [-]

Would love to see these numbers broken down by gender.

Comment author: Yvain 09 January 2014 12:12:09AM 8 points [-]

For the sake of simplicity, I used sex rather than gender and ignored nonbinaries. The average man on the site has a feminism approval score of 3.75; the average woman on the site has a score of 4.40. These are significantly different at p < .001.

The average man on the site has a social justice approval score of 3.55; the average woman on the site has a score of 4.21. These are, again, significantly different at p < .001.

Comment author: bramflakes 07 January 2014 03:29:23PM 3 points [-]

Offtopic, but ETA on the survey results being published?

Comment author: Yvain 09 January 2014 12:06:56AM 6 points [-]

Probably before the end of this month.

Comment author: jaibot 07 January 2014 04:57:15AM *  4 points [-]

~~Update: Likely that feminist-inclined LWers are less likely to comment/vote and more more likely to take surveys.~~

Meta-update: This hypothesis ruled highly-improbable based on more data from Yvain.

Comment author: Yvain 09 January 2014 12:06:17AM 9 points [-]

Among lurkers, the average feminism score was 3.84. Among people who had posted something - whether a post on Main, a post in Discussion, or a comment, the average feminism score was 3.8. A t-test failed to reveal any significant difference between the two (p = .49). So there is no difference between lurkers and posters in feminism score.

Among people who have never posted a top-level article in Main, the average feminism score is 3.84. Among people who have posted top-level articles in Main, the average feminism score is 3.47. A t-test found a significant difference (p < .01). So top-level posters were slightly less feminist than the Less Wrong average. However, the average feminism of top-level posters (3.47) is still significantly higher than the average feminism among women (3.1).

Comment author: ChrisHallquist 07 January 2014 03:36:38AM *  1 point [-]

...and the debate about whether Plan Y is or is not a good idea would immediately shift to a debate about whether Plan Y is or is not eugenics - which you would lose, because it is.

This assumes words have true, immutable meanings, which they don't.

For example, if you respond to the claim that Obama is a communist by...

Wait a second, you yourself explained this pretty well in your Anti-Reactionary FAQ: "The meaning of words changes over time, and the Cold War made the more moderate elements of communism drop the 'communist' label."

Accusations of "eugenics" generally deserve a similar response: meanings of words change over time, and the fact that a policy fits Galton's original definition of "eugenics" doesn't mean it's "really" eugenics any more than we should examine pre-Cold War communist documents to establish whether America is "really" a communist country.

Yes, arguing about definitions is annoying, but there's really no way around having to explain that no, policy Y is not what people commonly associate with "eugenics." The real choice you have is whether to alienate a portion of your audience from the outset by declaring you like eugenics.

And yes, ballsy countersignalling can sometimes work, but this doesn't mean it's automatically a good idea in every case. For example, I predict that politician who described her support for legal abortion by saying "I support murder in some cases" (*cough cough*) would have a very difficult time getting elected.

Comment author: Yvain 08 January 2014 01:10:13AM 9 points [-]

I don't think it assumes words have immutable meanings, just that they have some conventional meaning. You are proposing that we turn the debate from "Is eugenics plan x a good idea?" to "Does plan x, which fits the current conventional meaning of eugenics, sound like eugenics to you?" Unless you can unilaterally change the conventional meaning of eugenics, then for your purposes the meaning might as well be immutable - your argument will fail. And not only do people show no signs of being willing to shift the conventional meaning of eugenics in a pro-eugenics direction, but they seem very willing to shift the conventional meaning of eugenics in an anti-eugenics direction whenever anyone asks.

I think probably things inside and outside the Overton Window require different strategies. "Ballsy countersignaling" might work differently for things outside the window than for things inside of it. I agree that the politician shouldn't call abortion murder.

Comment author: Yvain 06 January 2014 11:53:53PM *  90 points [-]

Since it has suddenly become relevant, here are two results from this year's survey (data still being collected):

When asked to rate feminism on a scale of 1 (very unfavorable) to 5 (very favorable), the most common answer was 5 and the least common answer was 1. The mean answer was 3.82, and the median answer was 4.

When asked to rate the social justice movement on a scale of 1 (very unfavorable) to 5 (very favorable), the most common answer was 5 and the least common answer was 1. The mean answer was 3.61, and the median answer was 4.

In Crowder-Meyer (2007), women asked to rate their favorability of feminism on a 1 to 100 scale averaged 52.5, which on my 1 to 5 scale corresponds to a 3.1. So the average Less Wronger is about 33% more favorably disposed towards the feminist movement than the average woman (who herself is slightly more favorably disposed than the average man).

I can't find a similar comparison question for social justice favorability, but I expect such a comparison would turn out the same way.

If this surprises you, update your model.

Comment author: ChrisHallquist 06 January 2014 06:40:27PM -2 points [-]

The word isn't the whole problem, but this a case where not using the word would be painless and beneficial.

"Eugenics" is a problematic word because it's now associated with involuntary sterilization and Nazis. But for some reason, some supporters of voluntary human enhancement will go and use the term for things they support.

They can't control whether other people use "eugenics" to attack all kinds of things they don't like, but the least the former group could do is avoid actively aiding the latter group.

Comment author: Yvain 06 January 2014 11:17:31PM *  45 points [-]

Given that there is a popular tendency for people to accuse even totally different things of being "eugenics" to discredit them, if you tried to rebrand eugenics as something else people would notice very quickly, they would "accuse" you of being eugenicist, and the debate about whether Plan Y is or is not a good idea would immediately shift to a debate about whether Plan Y is or is not eugenics - which you would lose, because it is.

This reminds me of an interesting analysis I heard about why Heartiste manages to hang on when many people who are much less horrible than he is get laughed off the Internet. If you write some very reasonable liberal enlightened essay about how maybe there's some reason to believe some women are such-and-such but we must not jump to conclusions, people will call you a sexist, you'll have to argue that you're not a sexist, and your opponents have spent their entire lives accusing people of sexism and are better at this argument than you are and will win (or at least reduce your entire output to defending yourself). If you're Heartiste, and people call you sexist, you can just raise an eyebrow, say "Well, yeah", and watch people whose only master-level argumentative gambit is accusing people of sexism have no idea what to do

Heartiste happens to be awful, but I'm pretty sure the same strategy could be applied to reasonable positions. If I wanted to start a site that promoted sexist positions, I would call it www.sexism.com.

If the human biodiversity people had called their movement neo-racism, they would have avoided having every mention of them devolve into painful non-debates like this one. Compare the neo-reactionaries, who are much more politically astute and who were entirely correct to call their movement neo-reaction.

Comment author: passive_fist 01 January 2014 08:43:47PM 0 points [-]

The explanation offered on the page you linked seems more psychological (i.e. "I want to return to the weight I'm used to") rather than having any basis in the biological needs of the body. Is this assessment correct?

Comment author: Yvain 04 January 2014 04:35:27PM 1 point [-]

That wasn't my impression. See for example the paragraphs about the rats given foods of different caloric densities.

Comment author: Psy-Kosh 03 January 2014 06:07:53PM 0 points [-]

Well, I could bring a few extra chairs if wanted. (Although are we even still on for tomorrow given how the roads are? (Admittedly, sunday will probably be worse...))

Comment author: Yvain 03 January 2014 08:52:19PM 0 points [-]

We're still on for whoever can make it. I was out driving today and roads seemed a little better.

Comment author: Psy-Kosh 28 December 2013 07:11:59PM 1 point [-]

As of now, I'm planning on coming.

Anything I should be bringing? (ie, extra chairs, whatever?)

Comment author: Yvain 03 January 2014 02:22:59AM 0 points [-]

I think we're good, as long as some people don't mind sitting on the floor.

Comment author: passive_fist 29 December 2013 09:30:59PM 0 points [-]

I haven't read his book, but how does he define this set-point?

Comment author: Yvain 01 January 2014 12:01:27AM *  1 point [-]

I don't exactly remember, but I think it's whatever is a healthy weight for your height/age, plus or minus a genetic factor. And drugs/diseases/diets that make people obese do so by disrupting the set point or the body's ability to conform to the set point.

See also here

Comment author: Yvain 29 December 2013 04:11:42PM *  13 points [-]

Reminder: You have two days left to take the 2013 Less Wrong Census/Survey

Comment author: passive_fist 29 December 2013 09:35:05AM 1 point [-]

Since people generally eat a constant number of calories a day and since calories come from fat, protein, or carbohydrates, effectively mainstream nutrition was telling people to eat more carbohydrates.

Generally, when people restrict the type of foods they eat, they wind up eating less calories, so this statement is not entirely valid. As far as I recall, mainstream media was trying to get people to eat less fat and eat the minimum amount of protein needed for sustenance (which is arguably low but that's another debate entirely). If people had followed this advice, the total number of calories would be reduced.

Comment author: Yvain 29 December 2013 03:26:17PM 9 points [-]

Here I think you genuinely disagree with Taubes, who believes that people have a caloric set point of sorts and that they are never going to actually decrease the calories they take in, especially if they're not really trying to do so and just pursuing a separate goal like "cut down on fat".

Comment author: Yvain 25 December 2013 05:49:15AM *  74 points [-]

All right, I'll look through my old stuff later this week, find a very few embarrassing or controversial things I want to hide, and unlock the rest.

Comment author: Yvain 28 December 2013 07:07:52PM 17 points [-]

I've unlocked my old blog minus five or ten private articles. You should be able to read the rest. If you can't, let me know.

Comment author: Yvain 28 December 2013 07:04:09PM *  21 points [-]

Taubes' concern as I remember it was that mainstream nutrition was trying to get people to eat less fat at the same time they were trying to get people to eat less meat (ie protein). Since people generally eat a constant number of calories a day and since calories come from fat, protein, or carbohydrates, effectively mainstream nutrition was telling people to eat more carbohydrates. This was accurately reflected in the food pyramid where carbohydrates form the base and are most strongly encouraged, and in guidelines that say diets should be 55–75% carbohydrate.

Your statements here sound pretty damning, but in the gestalt impression I remember from his book, I don't remember Taubes pushing a claim that the government/mainstream was pro-sugar so much as a claim that the government/mainstream was pro-carbohydrate. This is both much more plausible, and totally sufficient to form a foil to Taubes' carbohydrates-are-bad theory. The sugar angle seems like kind of a distraction here.

Comment author: HatCloak 24 December 2013 04:20:01PM *  2 points [-]

[Edit: My points still stand, but this isn't an issue worth fighting over. I've gone back and edited my post.]

Look for the bar at the right side of the page. Look down to the part where it says "Recent on Rationality Blogs". The current top link there is the same link as what I gave. I therefore disbelieve that he asked that he asked not to link to it from Less Wrong, or that he still supports such a request if he did make it, because Less Wrong itself links to it in the sidebar! If it is true that we shouldn't link to his new blog, shouldn't the site itself be abiding by that as well? I trust the administrators of Less Wrong to not go against Yvain on that.

Comment author: Yvain 25 December 2013 05:54:02AM 10 points [-]

Speaking of the sidebar, is there any way to make it optional whether a post goes up there? Maybe by including [LW] in the title or something? I enjoy blogging about rationality, but I also enjoy blogging about random things that go on in my personal life, and it's kind of embarrassing to have those show up on the LW sidebar.

Comment author: HatCloak 24 December 2013 02:43:24PM *  20 points [-]

Edit: Yvain, the great guy that he is, is handling this amazingly. In my eyes, consider this resolved.

(This is an anonymous nick for the moment, but this issue needs to be raised and I'm not comfortable at this point doing it publicly under my own name.)

tl;dr as provided by Daniel_Burfoot: "Yvain is awesome, it's a shame he locked up his old stuff, let's lobby him to open it back up". I heartily endorse this summery, and it pretty much sums up what I have been saying.

[Due to some remarks, have redacted the links to Yvain's blogs, old and new. This is absurd, in my opinion. Yvain's new blog is a Rationality Blog in the Recent on Rationality Blogs part of the sidebar, and his old livejournal blog is linked to in many of his old posts. So I do not think that it is even meaningful to redact them. However, in the interest of not inciting argument, I have redacted them regardless.]

All of us here know of Yvain. He has posted much great stuff both here on Less Wrong, and on his blog. Insightful, brilliant stuff. If you go and look at the list of top rated Main posts, Yvain's stuff top's the lot.

A year ago, he switched blogs, from [old blog] to [new blog]. Well and fine. He had good reasons for doing so, including a desire for relative anonymity.

This, I do not object to.

However, as of a couple of weeks age, he locked his old blog, locking away many brilliant essays on a wide variety of topics. I view this as bad, to say the least. So much beautiful and brilliant stuff, of great interest to anyone interested in Less Wrong type stuff, never to see the light of day again.

What solutions are there? There are a few. My favorite so far is for Scott to restrict access by LW karma, which would allow him to maintain his privacy against the web, while still not denying those brilliant, humorous, and insightful posts to those who would truly appreciate them.

Or maybe you have another, better, idea? Please, suggest it. And whether or not you do, please, if you value the amazing contents of [old blog], help petition Scott to consider finding a solution. We have to be able to reach some sort of compromise, right?

There has been discussion in the comments section here [link removed; but it is currently the top post in the "Recent on Rationality Blogs" section] about this. Please, weigh in your support.

Comment author: Yvain 25 December 2013 05:49:15AM *  74 points [-]

All right, I'll look through my old stuff later this week, find a very few embarrassing or controversial things I want to hide, and unlock the rest.

Comment author: Yvain 18 December 2013 11:45:02PM 27 points [-]

In light of the unusual 3x offer, I have borrowed from future-Yvain's expected contributions to MIRI and donated $5000. It should qualify for matching.

Comment author: Yvain 09 December 2013 07:30:27PM 20 points [-]

Thank you for taking our ability to dismiss experts we don't like up one meta-level.

Comment author: Yvain 23 November 2013 06:55:52PM 11 points [-]

I just realized I forgot a very important question I really want to know the answer to!

"What is your 90% confidence interval for the percent of people you expect to answer 'cooperate' on the prize question?"

I've added this into the survey so that people who take it after this moment can answer. If you've taken the survey already, feel free to record your guess below (if you haven't taken the survey, don't read responses to this comment)

Comment author: radical_negative_one 22 November 2013 10:43:40AM *  41 points [-]

Survey completed in full. Begging for karma as per ancient custom.

I choose DEFECT because presumably the money is coming out of CFAR's pocket and I assume they can use the money better than whichever random person wins the raffle. If I win, I commit to requesting it be given as an anonymous donation to CFAR.

EDIT: Having been persuaded my Yvain and Vaniver, I reverse my position and intend to spend the prize on myself. Unfortunately I've already defected and now it's too late to not be an asshole! Sorry about that. Only the slightly higher chance of winning can soothe my feelings of guilt.

Comment author: Yvain 22 November 2013 10:06:13PM 23 points [-]

The money is coming out of my pocket, it is not funging against any other charitable donations, and I am in favor of someone claiming the prize and using it to buy something nice that they like.

Comment author: DanArmak 22 November 2013 09:50:22AM *  41 points [-]

Notes taken while I answered.

What is your family's religious background, as of the last time your family practiced a religion?

We're Ashkenazi Jews, but AFAIK the last time any ancestor of mine practiced a religion was in my great-grandparents' generation. (And then only because I knew only one of them personallyh, so it's reasonable to assume at least one of the others could have been religious.) I get that every human is descended from religious ones, but conflating this datapoint with someone whose actual parents practiced a religion once seems wrong.

Probability

For some of these my confidence was so low that I didn't answer. For some questions, there are also semantic quibbles that would affect the answer:

  • Supernatural: AFAIK there is no agreed-on definition of "supernatural" events other than "physically impossible" ones which of course have a probability of 0 (epsilon). OTOH, if you specify "events that the average human observer would use the word 'supernatural' to describe", the probability is very high.
  • Anti-Agathics: what counts as reaching an age of 1000 years? Humans with a few patched organs and genes? Cyborgs? Uploads with 1000 subjective years of experience?
  • Simulation: this is complicated by ontological differences: whether, when universe A is simulated in universe B, this somehow contributes to B's "realness" measure, or actually creates B. Is existence of a universe a binary predicate? I answered as if it is.

Type of global catastrophic risk: although I chose the most probable, there wasn't a large difference in estimated probability for the top few leading dangers.

about how often do you read or hear about another plausible-seeming technique

At first I thought "every few days". But then I realized these techniques almost never work out or are unsupported by evidence, and so it would be wrong to call them plausible-seeming. So I recalibrated and answered much more rarely.

Then I saw the next questions asked how often I tried the technique and how often it actually worked. But I already choose not to try them most of the time because I expect not to succeed. So I let my previous answer stand. I hope this was as intended.

CFAR bonus questions:

You are a certain kind of person

Are these questions claiming that I, DanArmak, am this kind of person who can change; or that everyone can change? The answers would be very different. I assumed the latter, but it would be nice to have confirmation.

Other nitpicks: a certain kind on which dimension? Some aspects of personality are much harder to change than others.

What is the measure of "true" change? By the means available to us today, we can't change into truly nonhuman intelligences, so does that mean our "kind" cannot be changed? And the answers to the questions will change over time as technology creates new more effective interventions.

And: does "basic things" mean "fundamental things" or "minor insignificant things"? Normally I would assume "fundamental things", but then it seems identical to the previous question.

On a personal note, this set of questions struck me as incompatible after answering the previous sets. They seem to deliberately probe my irrational biases and cached beliefs, and I felt I couldn't answer them while I was deliberately thinking reflectively and asking myself why I believed the answers I was giving.

How would you describe your opinion on immigration?

The politics of immigration in Israel are totally different from those of the US (and I expect this holds for many other countries too in their different ways). I didn't answer because I was afraid of biasing the poll, and it would have been nice to get more guidance in the question.

Comment author: Yvain 22 November 2013 10:03:57PM 4 points [-]

I endorse you still putting your background as Ashkenazi Jewish, as this gives interesting ethnic information beyond that in the race question.

Comment author: NancyLebovitz 22 November 2013 06:32:34PM 30 points [-]

I took the survey. Thanks for running it.

Should Muslim be divided into types?

I'm not sure what supernatural means for the more arcane simulation possibilities. I consider it likely that if we're simulated, it's from a universe with different physics.

I would rather see checkboxes for global catastrope, since it's hard to judge likelihood and I think the more interesting question is whether a person thinks any global catastrophe is likely.

Would it be worth having a text box for questions people would like to see on a future survey? I'm guessing that you wouldn't need to tabulate it,-- if you posted all the questions, I bet people here would identify the similar questions and sort them into topics.

Comment author: Yvain 22 November 2013 10:00:57PM 7 points [-]

So far no one of several hundred people has identified Muslim, so I think finer gradations there would be overkill.

I can't do checkboxes.

I ask every year what questions people want in a future survey on this site. That way the good ones can get updated and people can hold discussions about them.

Comment author: blacktrance 22 November 2013 05:50:01PM *  13 points [-]

I'm disappointed to see that most of my suggestions weren't used.

Comment author: Yvain 22 November 2013 09:58:45PM *  8 points [-]

I'm sorry. I couldn't put in checkboxes where you can choose as many as you want, because my software can't process them effectively. And I am reluctant to take suggestions about clarifying or adding more options to different questions as past experience has told me that no matter how fine the gradations are people always ask to have them finer. I took your suggestion about better divisions of Christianity and I thank you for making it.

2013 Less Wrong Census/Survey

78 Yvain 22 November 2013 09:26AM

It's that time of year again.

If you are reading this post, and have not been sent here by some sort of conspiracy trying to throw off the survey results, then you are the target population for the Less Wrong Census/Survey. Please take it. Doesn't matter if you don't post much. Doesn't matter if you're a lurker. Take the survey.

This year's census contains a "main survey" that should take about ten or fifteen minutes, as well as a bunch of "extra credit questions". You may do the extra credit questions if you want. You may skip all the extra credit questions if you want. They're pretty long and not all of them are very interesting. But it is very important that you not put off doing the survey or not do the survey at all because you're intimidated by the extra credit questions.

It also contains a chance at winning a MONETARY REWARD at the bottom. You do not need to fill in all the extra credit questions to get the MONETARY REWARD, just make an honest stab at as much of the survey as you can.

Please make things easier for my computer and by extension me by reading all the instructions and by answering any text questions in the simplest and most obvious possible way. For example, if it asks you "What language do you speak?" please answer "English" instead of "I speak English" or "It's English" or "English since I live in Canada" or "English (US)" or anything else. This will help me sort responses quickly and easily. Likewise, if a question asks for a number, please answer with a number such as "4", rather than "four".

Last year there was some concern that the survey period was too short, or too uncertain. This year the survey will remain open until 23:59 PST December 31st 2013, so as long as you make time to take it sometime this year, you should be fine. Many people put it off last year and then forgot about it, so why not take it right now while you are reading this post?

Okay! Enough preliminaries! Time to take the...

***

2013 Less Wrong Census/Survey

***

Thanks to everyone who suggested questions and ideas for the 2013 Less Wrong Census/Survey. I regret I was unable to take all of your suggestions into account, because of some limitations in Google Docs, concern about survey length, and contradictions/duplications among suggestions. I think I got most of them in, and others can wait until next year.

By ancient tradition, if you take the survey you may comment saying you have done so here, and people will upvote you and you will get karma.

Comment author: Dr_Manhattan 22 November 2013 03:33:52AM 44 points [-]

dude, no "jewish" religious background? seems like a serious omission unless my priors are all screwed up.

Comment author: Yvain 22 November 2013 03:37:33AM 19 points [-]

I'm sorry. I'm not sure how that happened. Must have accidentally gotten deleted when I was adding in the Eastern Orthodox stuff. The question has been fixed and "Jewish" is now an option.

Comment author: Vaniver 21 November 2013 02:59:13AM 3 points [-]

This doesn't tell us the root of the problem (why would feeling less sick go along with miscarriages?) but it does tell us cutting back on caffeine is probably not helpful.

Suppose everyone is subject to the same amount of environmental toxin, and some people react more strongly (by morning sickness, which vomits out those toxins). Then we would expect that the people less aversive to toxins would have more toxins, which would lead to more miscarriage.

In general, morning sickness seems to be good for pregnancy, but I don't think I've seen any evidence that it's adding good instead of removing bad (but, it would just be weird if it added good, so evidence may have been overlooked).

Comment author: Yvain 21 November 2013 10:04:24AM 3 points [-]

There's a very interesting study that finds that parents with more morning sickness have babies with better neurodevelopmental outcomes (20% had IQ > 130, compared to 7% of those with less morning sickness).

The researchers suggest that for some reason the hormone involved in producing morning sickness also aids child development. It's possible, but a better explanation is the one in your first paragraph - everyone gets exposed to toxins but people vary in how diligent they are in eliminating them, and extra diligence helps.

Comment author: Yvain 21 November 2013 09:53:56AM *  8 points [-]

What do the stats experts here think of "adjusting for confounders"?

I notice I often see correlational studies I would expect to be confounded by something (for example, a study showing that people who go to the doctor more often are less likely to get heart attacks might be confounded by income, since rich people can afford more doctor visits and also have different diets). Then the study says "We adjusted for confounders including income, class, race, and education, and the effect remained robust". Then they do a controlled experiment investigating the same thing and the effect disappears.

Is there any conventional wisdom in the stats community about how far to trust these kinds of adjustments?

2013 Census/Survey: call for changes and additions

27 Yvain 05 November 2013 03:10AM

I have finally gotten the survey to a point where I'm pretty happy with it. I have no big changes I want to make this year. But as is the tradition, please take a week to discuss what minor changes you want to the survey (within the limits of what Google Docs and finite time can do) and I will try to comply. In particular, we can continue the tradition that any question you request can be added to the Extra Credit section unless it's illegal or horribly offensive.

You can find last year's survey results here and you can find the very preliminary version of this year's survey (so far exactly the same as last year's) here.

EDIT: I don't particularly like the IQ test or the Big Five test used last year. If you have any better replacements for either, tell me and I'll put them in.

EDIT2: CFAR, you added seven questions last year. Let me know what you want to do with those this year. Keep them? Remove them? Replace them?

In response to comment by [deleted] on Open Thread, November 1 - 7, 2013
Comment author: gwern 03 November 2013 05:04:13PM 26 points [-]

If Yvain is (understandably) too busy to run it this year, I am willing to do it. But I will be making changes if I do it, including reducing the number of free responses and including a basilisk question.

Comment author: Yvain 04 November 2013 12:53:14AM 14 points [-]

Give me a few days to see if I can throw something together and otherwise I will turn it over to your capable hands (reluctantly; I hate change).

Comment author: Yvain 02 November 2013 07:01:08PM *  19 points [-]

Since high school I've been involved in conworlding - collaborative development of fictional worlds and societies, then setting stories or games in them.

Around 2005, I and some friends set a story in a culture with a goddess named Per married to a god named Elith. The religion gets called "Perelithve".

Skip to 2008. Neil Stephenson publishes Anathem, One throwaway reference mentions two of the avout, a woman named Per who marries a man named Elith. The marriage rite they invent gets called "Perelithian"

If names can have between 3 and 8 letters, and always alternate vowels and consonants, and ''th' counts as one sound, I calculate that the chances of someone who needs two names coming up with "Per" and "Elith" is on the order of one in a billion. The similarity in stories maybe adds another two or three bits of unlikelihood. If I've read 1000 novels, each of which has 100 minor characters, and my conworlds contain 1000 characters, then the odds aren't really that bad, maybe as high as 1%

Still freaks me out, though.

Comment author: Yvain 27 October 2013 07:33:16PM 1 point [-]

If you change the value of "medium" from "1" to "-5" while leaving the other two states the same, your conclusion no longer holds. For example, on your last graph, (very good, medium) would outrank (very good), even though the former has a value of -2 and the latter of +3. This suggests your system doesn't allow negative utilities, which seems bad because intuitively it's possible for utility to sometimes be negative (eg euthanasia arguments).

Comment author: Yvain 11 August 2013 07:39:44AM *  39 points [-]

I like this idea, but dislike inflation of the word "debunking".

Debunking means something was bunk and has now been conclusively proven wrong .Homeopathy has been debunked, creationism has been debunked, ESP has been debunked.

But when people say things like "Haven't you heard Searle debunked materialism?" or "Here's a link to an argument debunking Obamacare" it seems kind of like epistemological arrogance. It's not just "I disagree with you", but "There is no other side to this, it is now disproven in the same sense creationism is disproven and we can all go home."

I sort of accept the Myers-Briggs link as a debunking, because that fits the central category of "supposedly scientific theory that in fact has very poor support". The others seem more like controversial philosophical or political arguments. They're all really good controversial philosophical/political arguments I agree with, but I bet by the time this list reaches twenty entries some of them won't be.

I admit I don't have a better phrase. "Skeptical Argument Repository"?

Comment author: Yvain 08 August 2013 07:51:27AM *  39 points [-]

None of these are incorporated in molecular biology books and publications that I can find. But the answer was still there: visualize what I read. But not just visualize like the little diagrams of cellular interactions books usually give you – like stupid, over-the-top, Hollywood-status visualization. I had to make it dramatic. I had to mentally reconstruct the biology of a cell in massive, fast, and explosive terms.

I'm having the same problem with molecular biology right now, and I agree with the track you're taking. The issue seems to be the large amount of structure totally devoid of any semantic cues. For example, a typical textbook paragraph might read:

JS-154 is one of five metabolic products of netamine; however, the enzyme that produces it is unknown. It is manufactured in cells in the far rostral region of of the cerebrum, but after binding with a leukocynoid it takes a role in maintaining the blood-brain barrier - in particular guiding the movements of lipid molecules.

I find I can read paragraphs like this five or six times, write them on flashcards, enter them into Anki, and my brain still refuses to understand or remember them after weeks of trying.

On the other hand, my brain easily remembers vastly more complicated structures when they're loaded with human-accessible meaning. For example, just by casually reading the Game of Thrones series, I know an extremely intricate web of genealogies, alliances, locations, journeys, battlesites, et cetera. Byte for byte, an average Game of Thrones reader/viewer probably has as much Game of Thrones information as a neuroscience Ph.D has molecular biology information, but getting the neuroscience info is still a thousand times harder.

Which is interesting, because it seems like it should be possible exploit isomorphisms between the two areas. For example, the hideous unmemorizable paragraph above is structurally identical to (very minor spoilers) :

Jon Snow is one of five children of Ned Stark; however, his mother is unknown. He was born in a castle in the far northern regions of Westeros, but after binding with a white wolf companion he took a role in maintaining the Wall - in particular serving as mentor to his obese friend Samwell.

This makes me wonder if it would be possible to produce a story as enjoyable as Game of Thrones which was actually isomorphic to the most important pathways in molecular biology. So that you could pick up a moderately engaging fantasy book - it wouldn't have to be perfect - read through it in a day or two, and then it ends with "By the way, guess what, you now know everything ever discovered about carbohydrate metabolism". And then there's a little glossary in the back with translations about as complicated as "Jon Snow = JS-154" or "the Wall = the blood-brain barrier". I don't think this could replace a traditional textbook, but it could sure as heck supplement it.

This would be very hard to do correctly, but I'd love to see someone try, so much so that it's on my list of things to attempt myself if I ever get an unexpectedly large amount of free time.

Comment author: Asymmetric 17 July 2013 10:48:12PM 3 points [-]

Want to go; can't. Any future meetups planned outside of this one?

Comment author: Yvain 21 July 2013 05:28:38PM 0 points [-]

Not yet, but we'll see how this one goes and get back to you.

Meetup : Southeast Michigan

8 Yvain 16 July 2013 02:50AM
WHEN: 21 July 2013 02:00:00PM (-0400)

WHERE: [information removed]

I now live in Michigan and would like to get to know the locals. By a happy coincidence, this weekend out-of-staters ozymandias_42 and lucidian are also visiting. So I am inviting the Metro Detroit/Ann Arbor LW community to my house for snacks and informal discussion.

Sorry for the non-neutral location, but arundelo says last time everyone tried to meet up at a restaurant and it was loud and inconvenient and turned everyone off from further meetups. There are various people on here who can vouch for me being safe to visit. RSVPs appreciated but not required.

Comment author: Epiphany 28 June 2013 07:58:53AM *  3 points [-]

P/S/A: There's a treatable genetic mutation that half the population has which has more or less recently begun to be treated called MTHFR that causes several vitamin deficiencies (due to you not processing them into the usable forms - and it's treatable because you can take the usable form as a supplement) and homocysteine issues, and it's symptoms can range between none to raging horrible problems with depression, anxiety, IBS, fatigue, and a list of other things.

Specifics:

It reduces the body's ability to convert folic acid into the usable form, methyl folate and reduces the body's ability to convert vitamin B12 into the usable form (called methylcobalamin). This same mutation also tends to cause homocysteine levels to be too high or too low.

Caution:

Knowledge about this is kind of new, because we only mapped the genome so long ago (and figured out what this gene does, and figured out how to treat it, and began producing the supplements to treat it, etc). It can be tricky to treat. If you pursue this, you should seek a medical professional who has significant experience treating people with MTHFR.

What are the symptoms:

"Research is still pending on which medical conditions are caused by, or at least partially attributed to, the MTHFR gene mutations. From the partial list I recently went through on Medline, these are the current symptoms, syndromes and medical conditions relating to the MTHFR gene mutations" - www.mthfr.net This site lists 64 different conditions and symptoms ranging from miscarriages to schizophrenia. See Also: Disclaimer.

Disclaimer:

There's a reason I chose the symptom link above, but you should know that it is not a perfect list of symptoms. For an alternative list and an explanation about why I chose this symptom list, please see my response to Yvain about that under "the guy you're linking to".

Comment author: Yvain 29 June 2013 03:21:15PM *  15 points [-]

This is okay science, and unlike most times people link to these kinds of things on here I'm not going to throw a fit. But a few caveats:

First of all, my totally unfounded opinion is that the guy you're linking to seems sketchy as hell. His site implies he is a doctor a bunch of times, including his biography talking about when he entered medical school, but the only degree listed is one from a college of "naturopathic medicine". He founded a sketchy online pharmacy that sells (among other things) homeopathic solutions. He believes that vaccinations cause autism, at least in MTHFR babies (aka 1/4 of the population). And he fits a very worrying stereotype of the doctor who prescribes the same cure for almost every disease, and recommends that if it doesn't work you just need to "optimize" his cure a little more carefully, as opposed to consider that other factors may be involved.

I don't actually know anything about the research on this subject, but because of the red flags raised above I've tried to investigate it very briefly and see what it looks like. You should probably ignore everything below, but just out of curiosity:

Dr. Lynch seems to think that if you have MTHFR, taking more folic acid will be harmful to you. The way I've heard it is that taking more folic acid compensates for the lowered activity of MTHFR (see for example Linus Pauling Institute. Studies have also shown that low blood folate raises homocysteine levels.

If that's correct, then supplementing with MTHF would have similar effects to supplementing with normal folate - ie a mixed bag that seems to include higher risk of many cancers.

And although conventional wisdom is that homocysteine is a harmful amino acid that causes cardiovascular disease, this has been confirmed in correlational studies only and interventional studies have failed to show that decreasing homocysteine lowers heart attack risk.

My guess is that most of the disease associations Dr. Lynch mentions as linked to MTHFR are actually linked to folate in general, and people with MTHFR are more likely to have low folate and therefore disease. For example, Dr. Lynch mentions that MTHFR affects autism risk, Down Syndrome risk, neural tube defect risk, etc, but folic acid in general also affects all of these things. Likewise, this study shows that babies with mutant MTHFR have more autism, but that this effect only holds true in countries without folic acid supplementation - the implication being that in countries with folic acid supplementation, pregnant women have enough folic acid whether they have a gene that decreases their levels slightly or not.

I admit I have tried to read the important parts of his site but I haven't gotten to all of it, so maybe it explains his unconventional views on folate levels vs. MHTF levels somewhere.

Although Dr. Lynch recommends spending $150+ to get yourself genetically tested, my suggestion if you're worried about this or any other gene is to sign up for 23andMe and get all your common mutations sequenced in one go for $99. If you already have 23andMe data, you can find your version of the gene Dr. Lynch is talking about by clicking in the gear in the upper left hand corner, going to "Browse Raw Data", and then searching for SNP rs1801133. 23andMe uses opposite notation from most other sites, so the homozygous normal will be listed as GG, heterozygous as GA, and mutant as AA.

(I seem to be part of the lucky 25% who has the AA version. Maybe this explains my constant addictive infectious depressed demented diabetic cancerous heart attacks)

Your doctor can test you for folate deficiency. If you're really concerned about your health it's not a terrible idea, though it's no more important than about thirty other tests you could also get. If you have the mutant gene, maybe have a slightly more stringent standard than normal for what counts as deficiency? I don't know. In any case, you can supplement with MTHF if you want, but as mentioned before it will probably have about the same effect as normal folate supplementation - great for people with deficiencies and for pregnant women, probably just a little more cancer in everyone else. You can read more here.

The only exception is that if your psychiatrist recommends it for depression, you will need the MTHF version since the normal version can't directly enter the brain. Your doctor will recommend an expensive Official Medical Version of it. As far as I know, the MTHF you can get off Amazon for $15 works exactly as well.

I would be really interested in hearing from anyone who knows more about biochemistry or has studied these areas in more depth.

Comment author: coffeespoons 17 June 2013 02:49:03PM *  17 points [-]

Genes take charge and diets fall by the wayside.

You need a New York Times account to read it, but setting one up only takes a couple of minutes. Here are some exerpts in any case.

Obese people almost always regain weight after weight loss:

So Dr. Hirsch and his colleagues, including Dr. Rudolph L. Leibel, who is now at Columbia University, repeated the experiment and repeated it again. Every time the result was the same. The weight, so painstakingly lost, came right back. But since this was a research study, the investigators were also measuring metabolic changes, psychiatric conditions, body temperature and pulse. And that led them to a surprising conclusion: fat people who lost large amounts of weight might look like someone who was never fat, but they were very different. In fact, by every metabolic measurement, they seemed like people who were starving.

Before the diet began, the fat subjects’ metabolism was normal — the number of calories burned per square meter of body surface was no different from that of people who had never been fat. But when they lost weight, they were burning as much as 24 percent fewer calories per square meter of their surface area than the calories consumed by those who were naturally thin.

Thin people who are forced to gain weight find it easy to lose it again:

...His subjects were prisoners at a nearby state prison who volunteered to gain weight. With great difficulty, they succeeded, increasing their weight by 20 percent to 25 percent. But it took them four to six months, eating as much as they could every day. Some consumed 10,000 calories a day, an amount so incredible that it would be hard to believe, were it not for the fact that there were attendants present at each meal who dutifully recorded everything the men ate.

Once the men were fat, their metabolisms increased by 50 percent. They needed more than 2,700 calories per square meter of their body surface to stay fat but needed just 1,800 calories per square meter to maintain their normal weight.

When the study ended, the prisoners had no trouble losing weight. Within months, they were back to normal and effortlessly stayed there.

The body's metabolism changes with weight loss and weight gain:

The implications were clear. There is a reason that fat people cannot stay thin after they diet and that thin people cannot stay fat when they force themselves to gain weight. The body’s metabolism speeds up or slows down to keep weight within a narrow range. Gain weight and the metabolism can as much as double; lose weight and it can slow to half its original speed.

Genes and weight:

.A few years later, in 1990, Dr. Stunkard published another study in The New England Journal of Medicine, using another classic method of geneticists: investigating twins. This time, he used the Swedish Twin Registry, studying its 93 pairs of identical twins who were reared apart, 154 pairs of identical twins who were reared together, 218 pairs of fraternal twins who were reared apart, and 208 pairs of fraternal twins who were reared together.

The identical twins had nearly identical body mass indexes, whether they had been reared apart or together. There was more variation in the body mass indexes of the fraternal twins, who, like any siblings, share some, but not all, genes.

The researchers concluded that 70 percent of the variation in peoples’ weights may be accounted for by inheritance, a figure that means that weight is more strongly inherited than nearly any other condition, including mental illness, breast cancer or heart disease.

Comment author: Yvain 18 June 2013 07:10:26AM *  6 points [-]

On the other hand, here's a study that shows a very strong link between impulse control and weight. I'm not really sure what to believe anymore.

Comment author: Yvain 13 June 2013 11:31:16PM 14 points [-]

You mention Deep Blue beating Kasparov. This sounds look a good test case. I know that there were times when it was very controversial whether computers would ever be able to beat humans in chess - Wikipedia gives the example of a 1960s MIT professor who claimed that "no computer program could defeat even a 10-year-old child at chess". And it seems to me that by the time Deep Blue beat Kasparov, most people in the know agreed it would happen someday even if they didn't think Deep Blue itself would be the winner. A quick Google search doesn't pull up enough data to allow me to craft a full narrative of "people gradually became more and more willing to believe computers could beat grand masters with each incremental advance in chess technology", but it seems like the sort of thing that probably happened.

I think the economics example is a poor analogy, because it's a question about laws and not a question of gradual creeping recognition of a new technology. It also ignores one of the most important factors at play here - the recategorization of genres from "science fiction nerdery" to "something that will happen eventually" to "something that might happen in my lifetime and I should prepare for it."

Wikifying the blog list

32 Yvain 10 May 2013 08:18AM

Konkvistador's excellent List of Blogs by LWers led me to some of my favorite blogs, but is pretty well hidden and gradually becoming obsolete. In order to create an easily-update-able replacement, I have created the wiki page List of Blogs and added most of the blogs from Konkvistador's list. If you have a blog, or you read blogs, please help in the following ways:

-- Add your blog if it's not on there, and if it has updated in the past few months (no dead blogs this time, exceptions for very complete archives of excellent material like Common Sense Atheism in the last section)

-- Add any other blogs you like that are written by LWers or frequently engage with LW ideas

-- Remove your blog if you don't want it on there (I added some prominent critics of LW ideas who might not want to be linked to us)

-- Move your blog to a different category if you don't like the one it's in right now

-- Add a description of your blog, or change the one that already exists

-- Change the name you're listed by (I defaulted to people's LW handles)

-- Bold the name of your blog if it updates near-daily, has a large readership/commentership, and/or gets linked to on LW a lot

-- Improve formatting

Somebody more familiar with the Less Wrong twittersphere might want to do something similar to Grognor's Less Wrong on Twitter

Comment author: Eliezer_Yudkowsky 06 May 2013 07:55:45PM 2 points [-]

When we jump to the version involving causal nodes having Large leverage over other nodes in a graph, there aren't Large numbers of distinct people involved, but there's Large numbers of life-centuries involved and those moments of thought and life have to be instantiated by causal nodes.

(also, it seems less than 3^^^3-level certain that there's no clever trick to get effectively infinite computing power or effectively infinite computing time, like the substrateless computation in Permutation City)

Infinity makes my calculations break down and cry, at least at the moment.

Comment author: Yvain 06 May 2013 09:25:27PM *  14 points [-]

Imagine someone makes the following claims:

  • I've invented an immortality drug
  • I've invented a near-light-speed spaceship
  • The spaceship has really good life support/recycling
  • The spaceship is self-repairing and draws power from interstellar hydrogen
  • I've discovered the Universe will last at least another 3^^^3 years

Then they threaten, unless you give them $5, to kidnap you, give you the immortality drug, stick you in the spaceship, launch it at near-light speed, and have you stuck (presumably bound in an uncomfortable position) in the spaceship for the 3^^^3 years the universe will last.

(okay, there are lots of contingent features of the universe that will make this not work, but imagine something better. Pocket dimension, maybe?)

If their claims are true, then their threat seems credible even though it involves a large amount of suffering. Can you explain what you mean by life-centuries being instantiated by causal nodes, and how that makes the madman's threat less credible?

Comment author: Yvain 06 May 2013 07:36:26PM *  11 points [-]

How does this style of reasoning work on something more like the original Pascal's Wager problem?

Suppose a (to all appearances) perfectly ordinary person goes on TV and says "I am an avatar of the Dark Lords of the Matrix. Please send me $5. When I shut down the simulation in a few months, I will subject those who send me the money to [LARGE NUMBER] years of happiness, and those who do not to [LARGE NUMBER] years of pain".

Here you can't solve the problem by pointing out the very large numbers of people involved, because there aren't very high numbers of people involved. Your probability should depend only on your probability that this is a simulation, your probability that the simulators would make a weird request like this, and your probability that this person's specific weird request is likely to be it. None of these numbers help you get down to a 1/[LARGE NUMBER] level.

I've avoided saying 3^^^3, because maybe there's some fundamental constraint on computing power that makes it impossible for simulators to simulate 3^^^3 years of happiness in any amount of time they might conceivably be willing to dedicate to the problem. But they might be able to simulate some number of years large enough to outweigh our prior against any given weird request coming from the Dark Lords of the Matrix.

(also, it seems less than 3^^^3-level certain that there's no clever trick to get effectively infinite computing power or effectively infinite computing time, like the substrateless computation in Permutation City)

Michael Vassar in Europe

7 Yvain 24 April 2013 10:21PM

Michael Vassar, former president of the Singularity Institute and current Chief Science Officer of MetaMed, is currently visiting Europe and wants to meet up with Less Wrongers there. His schedule is:

25 April: Berlin

29 April: Estonia

8 May: London

12 May: Oslo

16 May: Nice (but may be able to meet people in Paris?)

26 May: Home to USA

If you have a meetup group in or near one of these cities, or you can put some people together, he's interested in talking about the Singularity, optimal philanthropy, and his work with MetaMed. You can reach him at michael.vassar[at]gmail.com

Comment author: Yvain 22 April 2013 05:08:48AM 1 point [-]

I can't make it this time, but I'll be moving to the Ann Arbor area in June and hope there will be more meetups then.

Comment author: Yvain 18 April 2013 06:46:53AM *  14 points [-]

Are you classifying 10% as a Pascal-level probability? How big does a probability have to get before you don't think Pascal-type considerations apply to it?

Are you suggesting that if there was (for example) a ten percent probability of an asteroid hitting the Earth in 2025, we should devote fewer resources to asteroid prediction/deflection than simple expected utility calculations would predict?

Comment author: yli 11 April 2013 08:25:43PM *  9 points [-]

I would have easily won that game (and maybe made a quip about free will when asked how...). All you need is some memorized secret randomness. For example, a randomly generated password that you've memorized, but you'd have to figure out how to convert it to bits on the fly.

Personally I'd recommend going to random.org, generating a few hexadecimal bytes (which are pretty easy to convert to both bits and numbers in any desired range), memorizing them, and keeping them secret. Then you'll always be able to act unpredictably.

Well, unpredictably to a computer program. If you want to be able to be unpredictable to someone who's good at reading your next move from your face, you would need some way to not know your next move before making it. One way would be to run something like an algorithm that generates the binary expansion of pi in your head, and delaying calculating the next bit until the best moment. Of course, you wouldn't actually choose pi, but something less well-known and preferably easier to calculate. I don't know any such algorithms, and I guess if anyone knows a good one, they're not likely to share. But if it was something like a pseudorandom bitstream generator that takes a seed, it could be shared, as long as you didn't share your seed. If anyone's thought about this in more depth and is willing to share, I'm interested.

Comment author: Yvain 11 April 2013 10:36:06PM 6 points [-]

When I need this I just look at the nearest object. If the first letter is between a and m, that's a 0. If it's between n and z, that's a 1. For larger strings of random bits, take a piece of memorized text (like a song you like) and do this with the first letter of each word.

Comment author: Yvain 11 April 2013 09:23:16PM 1 point [-]

50% chance I'll make it.

Comment author: Yvain 03 March 2013 11:31:06AM *  3 points [-]

I'm confused on how coordination works when there are two people coordinating the same thing at the same time.

Right now I have three people interested in playing in Berkeley, and I would prefer four. If you have exactly one person from your group who wants to play, I'd prefer you just send them my way. If there are five or more, then send one of them my way and keep five. If there are between one and five, we have something of a problem.

We should discuss this on IM.

Comment author: Yvain 02 March 2013 03:01:10AM 10 points [-]

I posted this in the waning days of the last open thread, but I hope no one will mind the slight repeat here.

The last Dungeons and Discourse campaign was very well-received here on Less Wrong, so I am formally announcing that another one is starting in a little while. Comment on this thread if you want to sign up.

Comment author: Yvain 27 February 2013 05:07:56AM 2 points [-]

The last Dungeons and Discourse campaign was very well-received here on Less Wrong, so I am formally announcing that another one is starting in a little while. Comment on this thread if you want to sign up.

Comment author: Yvain 22 February 2013 12:46:20AM *  28 points [-]

The Friendliness Problem is, at its root, about forming a workable set of values which are acceptable to society.

No, that's the special bonus round after you solve the real friendliness problem. If that were the real deal, we could just tell an AI to enforce Biblical values or the values of Queen Elizabeth II or the US Constitution or something, and although the results would probably be unpleasant they would be no worse than the many unpleasant states that have existed throughout history.

As opposed to the current problem of having a very high likelihood that the AI will kill everyone in the world.

The Friendliness problem is, at its root, about communicating values to an AI and keeping those values stable. If we tell the AI "do whatever Queen Elizabeth II wants" - which I expect would be a perfectly acceptable society to live in - the Friendliness problem is how to get the AI to properly translate that into statements like "Queen Elizabeth wants a more peaceful world" and not things more like "INCREASE LEVEL OF DOPAMINE IN QUEEN ELIZABETH'S REWARD CENTER TO 3^^^3 MOLES" or "ERROR: QUEEN ELIZABETH NOT AN OBVIOUSLY CLOSED SYSTEM, CONVERT EVERYTHING TO COMPUTRONIUM TO DEVELOP AIRTIGHT THEORY OF PERSONAL IDENTITY" or "ERROR: FUNCTION SWITCH_TASKS NOT FOUND; TILE ENTIRE UNIVERSE WITH CORGIS".

This is hard to explain in a way that doesn't sound silly at first, but Creating Friendly AI does a good job of it.

If we can get all of that right, we could start coding in a complete theory of politics. Or we could just say "AI, please develop a complete theory of politics that satisfies the criteria OrphanWilde has in his head right now" and it would do it for us, because we've solved the hard problem of cashing out human desires. The second way sounds easier.

Comment author: Luke_A_Somers 19 February 2013 06:00:31PM 1 point [-]

One of my only two errors on the christian side of year 2 was to suspect that a stereotypical Christian was a faker who was aiming for dead center. The other was an atheist who nailed the periphery.

So, the strategy of lying or selectively choosing topics to seem more typical within your group would not have worked on me.

I do think the whole 'I went to seminary' thing might best in the future be ruled out. It's one thing to create a fictional persona. It's another to give them a position of authority.

Comment author: Yvain 20 February 2013 12:06:25AM 2 points [-]

I don't even think it's about authority. Another person talked about how Christianity helped them through their drug addiction. Because there really are Christians who have been helped through drug addictions, but most contestants would have respected the spirit of the test too much to try the somewhat different exercise of making up a completely fake personality with a fake life history, this provided strong evidence of real Christianity.

Comment author: palladias 19 February 2013 02:35:47AM *  13 points [-]

The atheists and Christians were told to be honest when writing their own responses. So they shouldn't have been trying to game it in this way.

For year three, I've been thinking of doing just this:

I'd be interested in seeing differences between this test and one in which, say, Christians were just asked to discuss their opinions on some topics without it being part of a Turing Test, and then atheists were asked to fake Christian opinions on those same topics

On the topic of marriage, since people conceive of the institution of having really different purposes but usually get bogged down of the question of what laws should exist. I thought the question of "How should a couple decide whether to get married?" would provoke interesting responses.

Comment author: Yvain 20 February 2013 12:03:24AM *  11 points [-]

The atheists and Christians were told to be honest when writing their own responses. So they shouldn't have been trying to game it in this way.

"Honest" leaves a lot of wiggle room. If I were trying to write my honest atheist entry, what do I emphasize? That I hate scholastic philosophy and think religion set ethics back five hundred years? Or how I love C.S. Lewis and G.K. Chesterton and find many religious works to be among the most sublime creations of humankind? Both would be "honest".

Even if someone genuinely sets out not to present themselves at all, I still would expect presentation to be their main concern. There's a certain class of things which are impossible to do naturally. For example, if you try to count your natural respiratory rate, you will fail miserably; the fact that you're thinking about your breath immediately shifts it to consciously deciding what it is going to be. In my case, it makes it slower than normal. I can try to then consciously adjust by speeding it up, but since I don't know how much to speed it up, attempting to breathe naturally is basically just me trying to fake my natural breathing rate, probably badly.

I think self-presentation attempts of this sort raise some of the same problems.

Comment author: Yvain 19 February 2013 02:17:10AM *  28 points [-]

I'm having trouble determining the best strategy in these kinds of games, but I'm worried it's not quite actually sounding like a member of the group you're pretending to be.

For example, a liberal Christian complained that her (honest!) Christian answer did very poorly, because people associated liberalism with atheism. This suggests that the best strategy isn't necessarily to honestly list what you believe, but to list what you think a typical member of the group involved believes.

And If (for example) atheists know that the average Christian is writing about what they think the average Christian believes, than atheists in their fake entries will also write about what they think the average Christian believes.

Yes, if overdone, this is a sign of dishonesty; for example, anyone who was too stereotypical ("Yeah, I get up each day, read a selection from the Bible, check the Pope's Twitter account, then go to church, then go bomb an abortion clinic..." would be obviously fake.) So the best strategy seems to write something kind of stereotypical, but to depart from stereotype in a few places so as to signal you're talking about a real person rather than a straw man.

But this strategy is identical for real Christians and sham Christians, which sort of defeats the purpose of the Ideological Turing Test. We're not testing whether atheists can talk like a real Christian any more as much as whether atheists can talk like a real Christian pretending to be a stereotypical Christian, which seems like a lower bar.

I'd be interested in seeing differences between this test and one in which, say, Christians were just asked to discuss their opinions on some topics without it being part of a Turing Test, and then atheists were asked to fake Christian opinions on those same topics (also interested in how those same just-discuss entries would do against Christians-writing-for-a-Turing-test entries).

Interestingly, the entry that I was most convinced was Christian - and I was right - was one that included the phrase "and when I was in seminary...". I didn't expect any atheist to have the chutzpah to fake a priest, whereas I did expect some actual priests to read Leah's blog. This suggests that a winning strategy is to be stereotypical in unexpected ways fakers wouldn't think of, and possibly to be unstereotypical in unexpected ways fakers wouldn't think of (although obviously I can't think of any examples of this).

Comment author: Wei_Dai 16 February 2013 06:52:10AM *  32 points [-]

Why don't these writers post or at least cross-post on LW? I would really prefer that they did, for these reasons

  • It would give their posts more visibility and hence more comments and discussions. (I often learn more from the comments sections of a LW post than the post itself.)
  • I don't have to learn a new commenting system (get a new login and learn the markup/formatting, threading, and voting schemes).
  • I think the LW commenting system is generally better than that of any other blog I've seen.
  • If I comment on their posts, my comments can be backed up and searched along with all of my other LW comments.
  • I'm more motivated to comment (and to spend more effort on my comments) since my comments will be seen by more people, and I'm less worried about my comments disappearing when their blogs stop getting maintained.

Does it also have something to do with identity and affiliation? If so, maybe that's another reason to try to make people think of LW in less identity-related ways, or perhaps make the LW identity smaller / more inclusive somehow? (I don't know and I'd very much like to hear from one or more of these writers.)

Comment author: Yvain 16 February 2013 10:16:42AM *  45 points [-]

Less Wrong requires no politics / minimal humor / definitely unambiguously rationality-relevant / careful referencing / airtight reasoning (as opposed to a sketch of something which isn't exactly true but points to the truth.) This makes writing for Less Wrong a chore as opposed to an enjoyable pastime.

Comment author: satt 20 January 2013 03:12:35AM *  9 points [-]

I didn't bother reading Drum's article the first time I saw it circulating. I've known about Nevin's papers for a couple of years, already decided they were interesting but weak evidence, and Drum didn't seem to be bringing much new to the table. But I've now looked at his article, and it does reference stuff I wasn't aware of, like the city-level correlations between vehicular lead emissions and assaults.

We now have studies based on MRI scans, ecological correlations at multiple levels of aggregation, and longitudinal studies of individuals. But they're still all observational, and could still be affected by individual-level confounding factors. Qualitatively the causal mechanism is obviously real,* but it's only backed up by actual experiments in vivo for blood levels of 10-50 μg/dL (as far as I know). Drum's aiming an order of magnitude lower, where I'd still expect some effect, but I don't trust observational studies or simple extrapolation to estimate it precisely.

The very very obvious thing to do now is run a nice, big RCT. There are RCTs for lesser interventions (e.g.) but I see none for window replacement or soil cleanup.

So: you select 100 city neighbourhoods. Send an army of researchers to each of them to bang on every door and recruit as many families with infants & toddlers as they can. Send a second wave of researchers to measure the soil in the recruited kids' houses, take blood samples, administer the relevant IQ/development tests (test the parents too if you're feeling hardcore), and write everyone's demographic details (sex, age, race, etc.) down on their clipboards. Now you can choose 50 neighbourhoods at random as the treatment group; don't forget to check it's statistically comparable to the control group!

Having done that, get another load of people to bang on every door in the 50 neighbourhoods in the treatment arm, and offer them money to replace their windows and their soil. (I don't know whether you can get away with not replacing windows & soil for the residents who don't have kids in the study. If you could skip them you'd save a lot of money.) Then do the cleanup jobs.

That's not even the hard part. You now have to keep track of all the families you've recruited for the next 3 decades. A lot of them will move elsewhere. (Boosting the value of their houses by eliminating the lead might even encourage them to sell up.) That's fine — you just pay some academics to keep track of them. A year down the line, visit all of the families and run the tests again to measure the short-term changes. Some families will refuse the follow-up visit. That's OK too — you just keep their details on file, 'cause another 5 years later, when the kids are in school, you're gonna do a third wave of testing. Then a few more years later, when the kids are hitting puberty, you're gonna do a fourth wave of testing, and you're gonna get their criminal records from the cops. And again, a decade later, once they're adults, you do the testing and criminal record check again. And you might do another one down the line just to check there aren't any late-breaking effects to bite you. At each step, you compare the control & treatment subjects and publish the results.

How much might this all cost? Say there are 100 families' houses to clean in each of the 50 treatment neighbourhoods. That's 5000 houses. Drum says that Nevin says that replacing 16 million houses' windows would cost about $200 billion in total, or $12500 per house. Soil cleanup costs about as much again, making the cleanup total $25000 per house, or $125 million for the 5000 treatment houses. Add the cost of the blood testing, paying a few academics to run the study for decades, and other odds & sods, and the total cost might be something like $150 million. Oh yeah, and your study still wouldn't give you an unbiased estimate of the benefits of lead abatement; it'd underestimate them because of the time the children spent being exposed to lead in their houses before you showed up to run the study. But it'd give you a robust answer to a $400 billion question, and you could get part of that answer by the end of the decade.


* Blood lead poisoning has been observed since antiquity; sufficiently high exposure causes death within weeks or months; lead-treated lab animals show behavioural deficits; non-fatal but high blood lead levels cause obvious symptoms in children that can be partially reversed with chelation.

Comment author: Yvain 20 January 2013 03:48:46AM 14 points [-]

Having done that, get another load of people to bang on every door in the 50 neighbourhoods in the treatment arm, and offer them money to replace their windows and their soil. (I don't know whether you can get away with not replacing windows & soil for the residents who don't have kids in the study. If you could skip them you'd save a lot of money.) Then do the cleanup jobs.

A study sort of like this was done in Rochester, and they found that nothing they did changed blood lead levels very much and so they didn't learn anything from it. I guess they could go further with actually replacing people's houses.

Comment author: Yvain 19 January 2013 05:17:25PM *  31 points [-]

He says that myth is often taught resident-to-student, but it isn’t actually backed up by any research.

Lorber B, Swenson RM. Bacteriology of aspiration pneumonia. A prospective study of community- and hospital-acquired cases Ann Intern Med. 1974 Sep;81(3):329-31.

Brook I, Finegold SM. Bacteriology of Aspiration Pneumonia In Children, Pediatrics. 1980 Jun;65(6):1115-20.

Finegold SM. Aspiration Pneumonia Rev Infect Dis. 1991 Jul-Aug;13 Suppl 9:S737-42.

Bartlett JG. Anaerobic bacterial infections of the lung and pleural space Clin Infect Dis. 1993;16 Suppl 4:S248.

Yamashita Y et al. Anaerobic respiratory infection--evaluation of methods of obtaining specimens. Kansenshogaku Zasshi. 1994;68(5):631.

El-Solh AA et al. Microbiology of severe aspiration pneumonia in institutionalized elderly. Am J Respir Crit Care Med. 2003 Jun 15;167(12):1650-4. Epub 2003 Apr 10.

The kicker - if the doctor would stop to think about it, it should jump out as unintuitive – it would take some serious changes inside the lung to make an anaerobic infection – an infection of bacteria that thrive in areas with no oxygen. In reality it takes frequent aspirations over a long period of time to block off an area of the lungs.

The normal flora of the skin, mouth, and upper respiratory tract contains anaerobic bacteria; this alone should be sufficient to disprove overly simplistic "But there's oxygen, you can't have anaerobic bacteria there!" theories. Anaerobes are actually more common than aerobes in the mucus membranes of the mouth and URT and in fact outnumber aerobes in saliva about 10:1. According to The Anaerobic Microflora of the Human Body:

It is not surprising that anaerobes are present in large numbers in the flora of the intestinal and geintourinary tracts because oxygen concentrations are low in these regions. However, it appears unusual that they are also prominent members of the flora of the skin, mouth, nose, and throat - regions that are continuously exposed to air. The presence of anaerobes in these areas is explained first by the activity of the aerobic and facultatively anaerobic components of the flora that reside in association with the anaerobes and consume oxygen in their metabolism and second, by the colonization of anaerobes of microhabitats protected from air.

I think the moral of this story all people, be they doctors or kindergarteners, don’t usually check facts they’re taught, especially when being taught by an authoritative teacher.

Okay, here's what actually happened. In the 1970s and 1980s, a lot of people worked really hard studying the microbiology of aspiration pneumonia and all of them found lots of anaerobes. In the late 1990s, some other people, especially a guy named Paul Marik, tried the same thing using more modern techniques and found very few anaerobes. They concluded that the old studies had been wrong.

Some other people objected that anaerobes are really hard to detect and that maybe Marik and his supporters had just screwed up and not been able to find them even though they were there. This seems to be the view of Rene et al, who claimed to have repeated Marik's experiments using better technique and found lots of anaerobes just like the old theories would have predicted. It was then counterclaimed that Marik's experiment had been unusually rigorous and well-conducted, plus it was also easy to screw up the other way - that sometimes samples might have been contaminated by anaerobes in the upper respiratory tract that weren't responsible for the pneumonia at all. Everyone had a nice big fight about it which as far as I know still has not been fully resolved. UpToDate, which I tend to trust on this sort of thing, pushes the pro-anaerobe line, but emedicine, which is also pretty good, pushes the anti-anaerobe line. I do get the feeling the anti-anaerobe people now have the upper hand.

The rationalist thing to do would be to let the microbiologists fight it out among themselves and just study which antibiotics are most effective against aspiration pneumonia. The answer to that is very complicated, but the oversimplified answer as given by UpToDate is clindamycin, an antibiotic known for its efficacy against anaerobes, which seems to maybe suggest there was some kind of anaerobic component after all - but I am reading between the lines here on a subject I'm not really qualified to read-between-lines on and am probably completely wrong about this.

I agree that many doctors don't know anything about this (I hadn't heard of it until you brought it up and I checked the literature). The average doctor just checks every so often to see what antibiotic is recommended for aspiration pneumonia and then prescribes that one. As far as I know the antibiotic recommendations are still correct. This seems like a pretty efficient system, given how many things doctors have to know.

Anyway, as far as I can tell the real moral of this story is that medicine is really really hard and complicated and, like all science, often changes as technology improves and better experiments become possible. This is a less fun narrative than "Doctors are incredibly stupid and just by knowing about this one study I can totally outdo all of them" (YES, EVERYONE ON LESS WRONG, I AM TALKING ABOUT YOU) but fun narratives are wrong suspiciously often and this one is no exception.

Comment author: aaronde 15 January 2013 09:56:02PM 4 points [-]

http://www.youtube.com/watch?v=xiNvQ-g1XGs&list=PLDBBB98ACA18EF67C&index=19

This (admittedly biased) youtuber has a pretty thorough criticism of the study. The bottom line is that cholesterol tends to drop off before death (6:26 in the video), not just because cholesterol-lowering medications are administered to those at highest risk of heart attack (as Kawoomba points out), but also because of other diseases. When you correct for this, or follow people throughout their lives, this reverse causation effect disappears, and you find exactly the association you would expect: higher cholesterol associates with higher cardiovascular and total mortality (10:21).

I think that studies like this one are like studies showing that overweight is "protective" against mortality - when the obvious alternative explanation is that smoking, cancer, and other diseases can prevent weight gain, or cause weight loss, just before they kill you! Obviously, this would mask or even reverse the association between overweight (high cholesterol) and death, even if overweight (high cholesterol) causes death.

Comment author: Yvain 15 January 2013 10:12:38PM 9 points [-]

It's a good theory and the priors for it being true are high, but the one study that should have been able to test it directly got the opposite results as the theory would have predicted; patients with consistently low cholesterol over twenty years had higher mortality rate than patients with sudden drops in cholesterol.

One study isn't enough to draw any conclusions, but it does prevent me from considering the issue completely solved despite the elegance of this explanation.

Comment author: cypher197 01 January 2013 03:32:11AM 6 points [-]

Forgive me if I'm just being oblivious, but did anything end up happening on this?

Comment author: Yvain 01 January 2013 04:13:43AM 7 points [-]

I messaged Eliezer several times about this and he never got back to me. I talked to Tricycle, they said they were working on something, and what ended up happening was the split between Discussion and Main. This was not quite what I wanted, but given my inability to successfully contact Eliezer at the time I gave up.

Comment author: Yvain 25 December 2012 09:46:37PM *  1 point [-]

Sorry, read this wrong.

Comment author: Eliezer_Yudkowsky 20 December 2012 12:11:57AM 17 points [-]

1) In the long run, for CFAR to succeed, it has to be supported by a CFAR donor base that doesn't funge against SIAI money. I expect/hope that CFAR will have a substantially larger budget in the long run than SIAI. In the long run, then, marginal x-risk minimizers should be donating to SIAI.

2) But since CFAR is at a very young and very vital stage in its development and has very little funding, it needs money right now. And CFAR really really needs to succeed for SIAI to be viable in the long-term.

So my guess is that a given dollar is probably more valuable at CFAR right this instant, and we hope this changes very soon (due to CFAR having its own support base)...

...but...

...SIAI has previously supported CFAR, is probably going to make a loan to CFAR in the future, and therefore it doesn't matter as much exactly which organization you give to right now, except that if one maxes out its matching funds you probably want to donate to the other until it also maxes...

...and...

...even the judgment about exactly where a marginal dollar spent is more valuable is, necessarily, extremely uncertain to me. My own judgment favors CFAR at the current margins, but it's a very tough decision. Obviously! SIAI has given money to CFAR. If it had been obvious that this amount should've been shifted in direction A or direction B to minimize x-risk, we would've necessarily been organizationally irrational, or organizationally selfish, about the exact amount. SIAI has been giving CFAR amounts on the lower side of our error bounds because of the hope (uncertainty) that future-CFAR will prove effective at fundraising. Which rationally implies, and does actually imply, that an added dollar of marginal spending is more valuable at CFAR (in my estimates).

The upshot is that you should donate to whichever organization gets you more excited, like Luke said. SIAI is donating/loaning round-number amounts to CFAR, so where you donate $2K does change marginal spending at both organizations - we're not going to be exactly re-fine-tuning the dollar amounts flowing from SIAI to CFAR based on donations of that magnitude. It's a genuine decision on your part, and has a genuine effect. But from my own standpoint, "flip a coin to decide which one" is pretty close to my own current stance. For this to be false would imply that SIAI and I had a substantive x-risk-estimate disagreement which resulted in too much or too little funding (from my perspective) flowing to CFAR. Which is not the case, except insofar as we've been giving too little to CFAR in the uncertain hope that it can scale up fundraising faster than SIAI later. Taking this uncertainty into account, the margins balance. Leaving it out, a marginal absolute dollar of spending at CFAR does more good (somewhat) (in my estimation).

Comment author: Yvain 25 December 2012 06:08:05AM 12 points [-]

Thank you; that helps clarify the issue for me. Since people who know more seem to think it's a tossup and SIAI motivates me more, I gave $250 to them.

Comment author: SaidAchmiz 24 December 2012 05:24:57PM 3 points [-]

It makes sense that some people might be turned off by ritual. I hope those people went to one of the several other New York Less Wrong megameetups, or to the designated ritual-free day Sunday, or even on Saturday for the two hours before the ritual started. If they come to an event that has "RITUAL" in big letters all over it on the day when the ritual is scheduled to occur then I don't think you can fairly accuse it of being inflicted on them without such a sweeping redefinition of "consent" that it becomes impossible to ever do anything that doesn't exactly conform to social norms.

It was not my intention to accuse the ritual of being inflicted on anyone; I didn't think I said or implied such a thing, but if so, let me assure you that I quite realize that attendance was voluntary. As for the other megameetups, I will try to attend the next time a non-ritual one happens. I was sadly unable to make it on that Sunday. They seem to happen about once a year, yes?

Your other comments seem to suggest that you think that I am worried about brainwashing, or what have you; that's just not the issue here. So your comments such as

I don't know if it's possible to do inspiration without even the slightest chance of brainwashing, but I'd rather not ban all inspirational activities until we prove it.

To excise all of those things from our lives because we can't prove they don't cause some residual brainwashing would make the world less than it was.

miss the mark a bit. Like Risto_Saarelma, I just dislike rituals (fairly strongly). From your comment, and others in this thread, I've discovered that some (most?) people do like them, and like them enough to serve as motivation for traveling some distance, or at least for attending an event they'd otherwise skip. All I can say is: mind = blown. I really, genuinely did not expect this to be such a prevalent preference in the rationalist community.

Comment author: Yvain 25 December 2012 12:14:30AM 5 points [-]

I'm sorry, I may have either rounded you to the nearest cliche or lumped my responses to other people's comments into my response to yours.

Your comments about "social pressure" and "how can something be consensual if you enshrine it as a ritual" did make me think there was a consent aspect to it, and your comment about "using ritual to insert things deep into your psyche is something that I think is just bad" was where I got the feeling of brainwashing from, but I can see how I might've been misunderstanding them.

So you're saying you have such strong anti-ritual preferences that you assumed people must have been awkwardly attending something they didn't like in order to fit in? Hm. That makes sense.

I guess what I've learned from this is that I still can't describe the reasons for why I like things. "Community bonding" sounds good, but when you press me on it I admit it's kind of dumb and the ritual wasn't really about that at all. "Sense of the sacred" sounds good but there were a lot of other easier ways to get that feeling I didn't go for. I'm just going to say I have an unexplained preference for rituals of about the same magnitude as an unexplained preference for playing fantasy role-playing games, and although I can come up with just-so stories for it ("group bonding", "search for meaning", whatever) I can't explain it but would like to keep doing it anyway.

Comment author: Risto_Saarelma 24 December 2012 01:04:49PM 10 points [-]

The particular problem with the ritual is that unlike the other things, it seems to exist only for the purpose of community-building. Opting out of the other activities makes your cognitive dissonance module say "well, maybe I don't like fanfiction / board games / decision theory that much", which isn't that bad. Opting out of the ritual makes the cognitive dissonance module say "well, maybe I don't care that much for being a community member", which is a bit more unfortunate.

Then there's also the small thing where the nonsensical community forming rituals have popped up in every human culture everywhere as far back as we know anything about human cultures, and always tend to develop the side effect of the socially gelled people favoring each other a bit more over the boring people who don't bother to play along with the rituals. This is what the social instinct response is about, not paranoia about hooded murderers going about stabbing people one night. Traditional societies seem to end up with all members participating in whatever the local ritual is, because that's the guarantee for belonging in the in-group and the other in-group members having your back. If you don't see the point in the ritual, tough. The social cohesion mechanism wasn't built for you, just smile and play-act along for the bit of extra guarantee that someone might have food to spare for you as well on the next famine year.

Comment author: Yvain 24 December 2012 11:55:37PM 6 points [-]

I think that people's response to ritual is hard to explain. I have trouble explaining it. "Community-building" is sort of the default explanation. In the same way, if there were a real-life meetup, "getting to know people better", which is almost a synonym for "community building", is a default explanation because it's hard to explain why we like meeting people face to face when online conversation is so much easier.

I think you make a good point that presenting it as "community building" might sound exclusionary, and I will stop using that justification. But in the end I don't think it is any more about community building than meetups or board games or anything like that - only harder to explain, so that that explanation becomes more salient. Maybe we should just bill it as "Come and sing and feel emotions."

Comment author: Risto_Saarelma 24 December 2012 11:34:37AM 6 points [-]

I don't think I'm worrying about brainwashing concerns or failing the social proof respectability checks.

I'm seeing a community cementing thing that I see no intrinsic fun in and a lot of other people seem to be seeing intrinsic fun in, which is tripping my "pretend to be like the normals and make the extra effort to participate in the unfun thing that is actually fun for them because they are not like you and then they might be less likely to kill you" instinct.

Comment author: Yvain 24 December 2012 12:19:11PM *  11 points [-]

What if next year's ritual includes chanting thrice unto the heavens a solemn vow not to kill the people who don't go to next year's ritual?

...nah, I don't think anyone wants to to kill or even shun people who don't go to the ritual. Of the top ten LW contributors on the table on the right, only two of them (me and Alicorn) attended, and for me it was a last-second sort of thing. Eliezer didn't go. It would be kind of hard to shun Eliezer and 80% of the top contributors, even if people wanted to. Only a tiny proportion of the community went to the ritual and those who chose not to were in good company.

More philosophically, wouldn't the same complaint apply to having real-life meetups at all (wouldn't it exclude people who prefer to just talk via the Internet?) or writing HPMoR (now non-readers feel left out of a lot of discussions and don't get in-jokes, so they might feel pressure to read it) or CFAR minicamps (people who don't get selected to go might feel like they're less a part of the community). And I know one commenter above managed to find some trivial differences between board game night and ritual night, but the fundamental problem of "What if I don't enjoy this but I feel like I have to go anyway?" remains sound.

I think the road of not doing fun things that most people want because someone who doesn't want to do it might feel left out leads to sitting quietly in a dark room. And the road of never doing community cementing things because people might be outside them leads to never trying to cement the community. And I think that the ritual, in this case, is being held to a much higher standard than any other activity of this sort just because it sounds kind of weird.

Comment author: SaidAchmiz 24 December 2012 02:35:53AM 4 points [-]

Since you mention that you had the same reaction last year, I assume that you've read the discussion that took place then that covers this issue, yes?

Uh... maybe? I don't remember, honestly. Link?

You are talking as if a required ritual was sprung on unsuspecting meetup participants. This is obviously not true. Both Raemon's and my rituals were advertised specifically as such. If you don't want to participate in a ritual don't go to one.

Well, yes, obviously. I didn't mean to imply that anything was sprung on anyone, just saying that yes, I wouldn't go to such a thing. The issue for me is that this (for any value of "this"; the NYC LW group, for instance) is a group that does this sort of thing.

Again, this is a good route towards consensual group/self-hacking.

How can something be consensual if you enshrine it as a ritual? Once it's ritualized, it stops being consensual, except insofar as you can choose whether to go along with it or leave the group. Not participating is inherently alienating.

The ideas they [the litanies of Tarski and Gendlin] represent ... are ideas that I very much WANT to be put as deep into my psyche as I can get them.

Really? I rather think I want these things as deep in my psyche as is warranted by how true/useful I judge them to be, and absolutely no deeper than that.

In any case, it... doesn't seem like this is the only purpose of these rituals — and quite unlikely that it's the only effect.

we were specifically looking for ways to hack ourselves into feeling closer to the people we want to feel closer to.

I... don't understand this sentiment at all. That is, I don't understand what you mean by this (and consequently don't understand why it's something you'd want). Clarification would be much appreciated.

I know Raemon even made the solstice event a two-day event. One day was ritual, and the other was just a regular get-together. So if you wanted to meet everyone without taking part in the ritual you could just go to the second day.

From a comment on Yvain's review:

One thing I wish I had known was that if you skip the ritual and just come to the "general meetup" the next day -- as seemed like a perfectly good option on paper -- not only do you miss out on the fun described in the post, but you look and feel totally lame. Well, okay, maybe that's a slight exaggeration, since you still do get to meet (and re-meet) great LW (and LWish[*]) people; but those people will have just spent the previous day in, you know, a bonding ritual which, strangely enough, has this weird effect of bonding them together more closely to each other than to you who weren't there. (Also, they will have gotten most of their general meetup-socializing done the previous day, before the ritual, so as not to be strangers to each other during the latter.)

Seems like about what I would expect.

Comment author: Yvain 24 December 2012 10:12:24AM *  28 points [-]

It makes sense that some people might be turned off by ritual. I hope those people went to one of the several other New York Less Wrong megameetups, or to the designated ritual-free day Sunday, or even on Saturday for the two hours before the ritual started. If they come to an event that has "RITUAL" in big letters all over it on the day when the ritual is scheduled to occur then I don't think you can fairly accuse it of being inflicted on them without such a sweeping redefinition of "consent" that it becomes impossible to ever do anything that doesn't exactly conform to social norms.

A lot of the above criticisms act like the ritual ruined a perfectly good meetup, but I think without the ritual this meetup would not have occurred. I went there all the way from Maryland because I wanted to see the ritual after reading about it last year. I dragged a friend who was also there because she loved rituals. Many people there weren't even in the LW community at all and came only because they wanted to see what the ritual was about. Quite a few people organized cross-continental flights from California because they wanted to participate in the ritual. If Raemon had said "Okay everyone, let's have yet another meetup and talk about Bayes for a few hours", there simply wouldn't have been a meetup of this size.

So on utilitarian grounds, I think it is a pure loss to take dozens of people for whom this is one of the highlights of their year, and hold them hostage to the purely theoretical possibility that there might be someone who wants to go to a megameetup, refuses to go to any of the many non-ritual megameetups, mysteriously hates ritual despite her insistence on only going to the megameetup with "ritual" in the name, refuses to go on the designated non-ritual day, refuses to leave or even go upstairs when the ritual starts, and is such a utility monster that her needs outweigh those of everyone else.

And knowing what the ritual was like, it's also hard for me to take the idea of exclusion seriously. I understand why people who came on the second day might feel like they missed something, but the same would be true if we'd all gone out to watch a movie Saturday night and they hadn't seen it. But other than that? We sung silly songs from Portal and Firefly together for a while. Now we are closer far than brothers, and shall never again feel respect or human feeling for anyone outside our sacred circle.

No, seriously, I do understand the idea of the Dark Arts (I coined our community's use of the term). But I think there's a difference between brainwashing and inspiration. Brainwashing changes your beliefs, inspiration helps you live up to them. I don't know if it's possible to do inspiration without even the slightest chance of brainwashing, but I'd rather not ban all inspirational activities until we prove it.

If it helps, think of this less as people in robes chanting in a dead language and more as our version of a school graduation (which, uh, also involves people in robes chanting in a dead language, but you get the picture). At a school graduation they sing songs, somebody talks about the values of the school and the importance of learning, and then everyone goes forth psyched about their future and has fond memories later. The same is true of weddings, funerals, the Fourth of July, birthday parties, summer camps. To excise all of those things from our lives because we can't prove they don't cause some residual brainwashing would make the world less than it was.

As a secular Jew, I grew up treating Passover and even my Bar Mitzvah in about the same way I treated birthday parties and summer camps; a fun time to get together and sing and appreciate family and friends, a marking of life transitions and the passing of time. But it grew harder and harder for me to appreciate them when I realized that I didn't really approve of celebrating the deaths of the Egyptian first-born, or that chanting the Torah and drawing moral lessons from it felt kind of like BSing in front of everyone. Even school graduations got to feel a little like "go forth and be a cog in a slightly more complex machine than the one you are currently a cog in."

In New York, I was able to have a good time and sing and meet people and mark the passings of the seasons, and for one of the first times affirm values that I really believed in. When Raemon got up there and started talking about how each year the sun went dark, and each year people died during the winter, but each year the sun came back - and when he went onto how each year we humans added a little bit more of our own light, and one day we would conquer the darkness entirely and no one would have to die anymore - well, it was a beautiful experience. Not life altering, not rationality-maximizing, but beautiful. And to be terribly clinical, the person I was before the ritual would have approved of everything that went on, which is not a test simple brainwashing can pass.

I agree that experiencing the sacred can be done individually. So can sex. It's not the same, though, in either case. Ever since the days of lighting fires and chanting about the gods, we've been doing our sacredness in groups. Even the ancients who took "sacred" totally literally knew you needed a minyan. It just works better. It's something I feel a need for. And I'd only fulfill that need in a group I completely trust, because of the brainwashing and awkwardness concerns you mentioned. And this was it. I don't think I took it seriously on an intellectual level, and I didn't have the same feelings as the people above who said it helped cement concepts into their mind. But I felt catharsis afterwards. I smiled a lot and sang a lot and it was good.

A commenter downstream said that his worry here "has to do with seriousness and how much I value banter and puncturing self-importance". I think we already have too much self-importance puncturing; too much irony. I think on the scales of "funny things should be taken lightly, solemn things should be taken with solemnity", we are too good at the former and too worried about social stigma to do the latter. In fact, of everything I'm impressed with Raemon for, the thing that impressed me most was his ability to take himself seriously, to resist the overpowering urge to say "Ha ha guys, I don't really think I'm cool enough to have strong feelings and officiate a celebration of the seasons, it's all just ironic, I'm not silly or something."

Except when it was silly. Raemon had the rare skill to design a night for us that was both funny and solemn at once, in exactly the right places, and to the exact degree that each idea required. I appreciate the humor but I think it was the solemnity that was beautiful.

Comment author: SaidAchmiz 24 December 2012 01:12:01AM *  28 points [-]

I considered not posting this comment, because it seems like you guys (and Raemon especially) have put a lot of effort into this, but I do want to sort out my response to this whole thing. Please don't take this as judgment; I'd really like to hear input about whether my reaction is wholly unwarranted, and why or why not.

When I read about this event (and I had a similar reaction to reading about last year's one, too), I get a strong sense of "ick"; a deep and profound feeling of being creeped out. I mean, you're designing and instituting a ritual. Intentionally. Why on earth would you do something like that?

From Yvain's review:

The idea was that since most rationalists and Less Wrongers are atheists for whom the traditional categories of Christmas and Hanukkah don't apply (and, let's face it, way too white for Kwanzaa), we would make our own ritual,

Whyyyyyyyy????

Why not just get together and hang out and... I don't know. Play party games? Talk? Watch movies? Why a ritual?

one centered around rationalist ideals, and use it as a Winter Season Positive Affect Schelling Point the same way all the religions do theirs.

Ok, I thought I knew what a Schelling Point is, but this usage puzzles me a bit. If I'm interpreting it right, though, my question is... why do this?

Is it because some (most?) of the people involved come from religious backgrounds, and miss the holiday rituals that took place in their families?

By the way, what happens if people present don't want to participate in the songs? Is there social pressure? I know I'd feel pretty darn uncomfortable if I was at a gathering and everyone started a collective sing-along.

Separately and unrelatedly, I really feel rather unsettled by the fact that you're using Eliezer's writings as a kind of... I don't know, mass? Sermon? It seems to me like that's taking entirely the wrong message away from all of it... to actually enshrine it as a sacred tradition or ritual of some sort.

I lurk on the OB/LW NYC mailing list, and you guys seem like pretty interesting people (I've been to one or two of the "public" meetups that were like... 1-2 years ago, now?); once in a while I think that maybe I should try and come to some of your meetups on occasion.

Stuff like this pushes me away. That's probably unfortunate, so if someone from the group (or whoever, really) could explain this whole ritual business to me, I would appreciate it.

Comment author: Yvain 24 December 2012 02:34:33AM 2 points [-]

Ok, I thought I knew what a Schelling Point is, but this usage puzzles me a bit. If I'm interpreting it right, though, my question is... why do this?

I stole that usage from (I think) Marcello. The idea is that people come together to be thankful and have fun and celebrate in one specific place and time and manner, and that makes it a unifying event that everyone enjoys more.

In response to Against NHST
Comment author: Yvain 21 December 2012 08:32:21AM 9 points [-]

I think (hope?) most people already realize NHST is terrible. I would be much more interested in hearing if there were an equally-easy-to-use alternative without any baggage (preferably not requiring priors?)

Comment author: Yvain 18 December 2012 06:22:09AM *  19 points [-]

I have some money that I was saving for something like this, but I also just saw Eliezer's (very convincing) request for CFAR donations yesterday and heard a rumor that SIAI was trying to get people to donate to CFAR because they needed it more.

This seems weird to me because I would expect that with SIAI's latest announcement they have shifted from waterline-raising/community-building to more technical areas where CFAR success would be of less help to them, but I'd be very interested in hearing from an SIAI higher-up whether they really want my money or whether they would prefer I give it to CFAR instead.

In response to 2012 Survey Results
Comment author: Yvain 14 December 2012 08:18:46PM 4 points [-]

I didn't do this myself because I didn't trust my statistical ability enough, and I forgot to mention it on the original post, but...

Can someone check for birth order effects? Whether Less Wrongers are more likely to be first-borns than average? Preferably someone who's read Judith Rich Harris' critique of why most birth order effect analyses are hopelessly wrong? Or Gwern? I would trust Gwern on this.

Comment author: hyporational 14 December 2012 11:33:48AM *  6 points [-]

That was fast. This is why I love you and the rest of the community.

Before I say anything else, let me remind everyone of something. Atherosclerosis is a systemic disease. When we're talking about arterial disease, mortality is not the only endpoint we're interested in. Most of the time a cardiovascular event will not kill you, it will leave you disabled. It's also a hell of a painful way to die. A stroke very rarely kills you, but most of the time leaves you less functional. Microinfarctions in the brain will cause dementia, but you might not die of it. Atherosclerosis in the leg will first make you lose sensation and function in the leg, and later you might lose the whole leg. That will probably not be lethal either. It would of course be intellectually dishonest to say that these events are not correlated with mortality, however.

Since only citing things from Framingham or Cochrane is a hard constraint to keep I am forced to commit the minor sin of citing a work not published in a peer-reviewed journal and refer you to the Framingham Diet Study, a subpart of the Framingham Heart Study whose methodology was published appropriately but whose results for some reason weren't. A guy who tracked down the results reports on them here and finds that....

I'm sorry. I'm not trying to be extra difficult, but where is the original source? Is it authentic? (Edit: Here, unfortunately not accessible, thank you Yvain)

If you'll allow me to step out of Framingham and Cochrane for a moment I can also link to a review on egg consumption and LDL which found that in most of the population there's no relationship.

Permission granted ;) I accept that the effect of dietary intake varies between individuals. Even this review recognizes that there are "hyperresponders" to dietary cholesterol. I also think that for a motivated individual measuring their response to diet would be optimal compared to just blindly switching. Measuring lipid profiles and other risk factors is cheap. I'm not sure how to measure subtypes of LDL, however, and to be honest I know nothing about their clinical relevance.

You may also be interested in a Cochrane review that finds no effect reducing dietary fat consumption on cardiovascular events, cardiovascular mortality or total mortality, and finds that changing the type of fat slightly decreases cardiovascular events (look how close that confidence interval gets to 1!), but not cardiovascular mortality or total mortality.

As I said, mortality is not the only interesting endpoint. Also the CI upper bound for CVD is not over 1, not matter how hard we want to push it. The other review does support your conclusions. It doesn't however support increasing dietary saturated fat or changing nutritional guidelines in any other way.

The only lipid-lowering drugs that seem to be consistently awesome are, of course, the statins, but they seem to work equally well in high and low cholesterol populations leading some to think they also have non-cholesterol-related effects.

That statins work equally well in high and low cholesterol populations is to me the most interesting claim that you make. Can you provide a source for it? It is commonly accepted that they have benefits on top of them affecting lipids, but that the effect is completely isolated is contrary to my knowledge. The reason for the bonus effect is also a mystery. The other drugs you mention have common annoying side effects that mostly reduce compliance, and I wouldn't be surprised if some of the side effects increased mortality. In Finland they are also usually prescribed by a specialist and are never a first line treatment.

I also just think LDL is a lousy biomarker. A majority of heart attack patients have LDL levels considered normal; this is especially impressive considering the high results of high cholesterol even in the general population.

We have guidelines to measure lipid profiles after 48 hours of an ischemic vascular event. Within this time period, the LDL levels plummet, but then they rise again. Since this is a very recent guideline, it might explain the finding you present. Then again, it also might not. If I skimmed correctly there was no mention of the timing of the measurements. Look below for Edit2 for a better explanation.

Framingham found that mortality increased with increasing cholesterol in people younger than 50. In people older than 50 (ie 90% of heart attack victims) it found no relationship (other sources say low cholesterol led to higher mortality in these age groups, but I can't access the paper to check.)

Atherosclerosis is mostly a nonreversible progressive disease that can start as early as in late adolescence, so it makes sense that hypercholesterolemia before age 50 is most important for its development. All it takes for a plague to rupture after tens of years of accumulation is that the endothelium covering it fails, it doesn't necessarily have to grow anymore. (Edit: see additional explanations in the next comment) I'm definately more critical about these issues in older age groups and probably should read more about them. We're taught that statins are useful even in people over 75, but maybe this has nothing to do with cholesterol. If you can tell me what the other source is, I can look it up. I might have access ;) Edit: Apparently neither of us do.

Only standard labs are readily available to me in clinical practice. We have a mostly public health care system, and nonstandard labs are usually ridiculously pricey. Of course LDL is hardly the only measurement we take, and is combined with all the other risk factors when assessing total arterial disease risk.

I hope I have provided a POV of how background knowledge can change the way we interpret study findings, and how much easier it makes sceptiscism about them. I apologize that I don't have english sources ready at hand for the claims I make, and I know that you will not take them on authority. It is impossible to me to keep record of most of my sources, and most of them are in finnish language.

I will check out the book you recommend, I'm chillin' after all. I think there are far too few rationalists in medicine. The education methods are authoritative and many times frustratingly ineffective. Unfortunately I don't know how to change it (yet), and will do my best with what I have.

Comment author: Yvain 14 December 2012 07:52:01PM *  6 points [-]

I'm sorry. I'm not trying to be extra difficult, but where is the original source? Is it authentic?

If I understand correctly, the original source is this. I can't find any way to access it online, but everyone who talks about it says the results mentioned in that blog post and no one says otherwise, so I'm assuming they didn't just make it up.

As I said, mortality is not the only interesting endpoint. Also the CI upper bound for CVD is not over 1, not matter how hard we want to push it. The other review does support your conclusions. It doesn't however support increasing dietary saturated fat or changing nutritional guidelines in any other way.

I see where you're coming from. On the other hand, there is a $40 billion diet industry telling people to eat less fat and causing a spectacular amount of mental anguish around this idea (I don't know if this is true in Finland; it's definitely a big deal in America). That eating less fat has literally zero effect on any outcome seems like something we should be trying to make more widely known, whether or not the small effect of changing fat types on events but not mortality holds up.

That statins work equally well in high and low cholesterol populations is to me the most interesting claim that you make. Can you provide a source for it?

Apparently not; I know I've heard this but my attempts to Google a study in humans failed. I did find something in rabbits showing effects regardless of cholesterol, but even that wasn't a direct comparison.

The other drugs you mention have common annoying side effects that mostly reduce compliance, and I wouldn't be surprised if some of the side effects increased mortality. In Finland they are also usually prescribed by a specialist and are never a first line treatment.

Right, but the studies on niacin and ezetimibe showed that they decreased cholesterol (ie were being used successfully and correctly) but failed to decrease cardiovascular endpoints.

We have guidelines to measure lipid profiles after 48 hours of an ischemic vascular event. Within this time period, the LDL levels plummet, but then they rise again. This might be caused by a ruptured atherosclerotic plague releasing its contents in the blood stream. Since this is a very recent guideline, it might explain the finding you present. Then again, it also might not. If I skimmed correctly there was no mention of the timing of the measurements.

You're right that the measurements were taken within 24 hours, but I've heard this isn't such a big deal, and according to the full-text version of the Fonarow study (sorry, didn't find it until this morning) they agree with me. Also, if I'm reading their table right patients having acute coronary events had higher, not lower, cholesterol than those coming in with chronic complaints, and if the effect were really only 5-15% it wouldn't significantly affect the main finding of the paper by much.

If you can tell me what the other source is, I can look it up. I might have access ;)

It's the same Framingham paper. I can only reach the abstract (my medical school doesn't give me electronic access that far back) and I'm relying on reviews and comments to tell me what's inside.

Atherosclerosis is mostly a nonreversible progressive disease that can start as early as in late adolescence, so it makes sense that hypercholesterolemia before age 50 is most important for its development. All it takes for a plague to rupture after tens of years of accumulation is that the endothelium covering it fails, it doesn't necessarily have to grow anymore.

Right, this makes sense, I'm just saying it's the opposite of what Framingham shows. Framingham says that "there is a direct association between falling cholesterol levels over the first 14 years and mortality over the following 18 years (11% overall and 14% CVD death rate increase per 1 mg/dL per year drop in cholesterol levels"

I hope I have provided a POV of how background knowledge can change the way we interpret study findings, and how much easier it makes sceptiscism about them

Yes, I agree with this. But the thesis of Good Calories, Bad Calories is that this allows enough degrees of freedom to be able to back up infinite amounts of confirmation bias. That is, if you see a study that supports your hypothesis, you say "Great, studies have proven we're right!" And if you see a study that opposes your hypothesis, you say "in light of my background knowledge that my hypothesis is right, we can't take this study at face value", then seize on the first flaw you find in the study and use it to throw it all out. Kind of the "one person's modus ponens is the other person's modus tollens" thing people here keep talking about.

My new favorite study ever is the Biblically-named Lee, Lord, and Lepper, which asked people to evaluate the methodology of a study on the death penalty. They found that regardless of its actual methodology, if the experimenters wrote the conclusion such that it supported the opinions of the evaluators, the evaluators said it had good methodology. If the experimenters changed the conclusion so that it disagreed with the opinions of the evaluators, the evaluators were - surprise! - able to find a bunch of problems with the methodology and reasons why the study didn't apply to anything.

I have no idea to what degree that's happening in medicine these days; I'm really only beginning to seriously engage with the literature beyond a boring student level. I read Good Calories, Bad Calories on the advice of a bunch of other Less Wrongers, it was a really interesting book and has gotten me worried enough that I wanted to vent - as you pointed out, this wasn't the best place for it and I apologize.

But I do think that further investigation beyond the level of just agreeing we can use background knowledge to interpret away findings is necessary at this point. At the very least, you have to admit the Cochrane review showing restricting dietary fat had no effect on anything means that something has gone atrociously wrong somewhere between what doctors say to their patients and reality (actually, I don't know how that works in Finland; in the US dietary fat is a pretty big deal).

Comment author: hyporational 13 December 2012 03:15:29PM *  13 points [-]

I'm a fifth year med student from Finland, a long time lurker, and you provoked me to post the first time. Thank you for that, I'm a terrible procrastinator. I'm a also big fan of many of your submissions. The alcohol fact seems right and the other two I can't comment on, so I'll focus on cholesterol. In this case I'm surprised of seeing no criticism.

If the information you present about cholesterol is true, I am highly surprised too, and I hope you're right because otherwise the information is potentially very deadly. The effect, whatever the direction, is amplified by you being a high status member of this community. If there were no other comments on this, I wouldn't bother either.

I would like to see the sources that REALLY changed your mind, and the link you provide doesn't seem to contain any good sources at a glance. The post that Kresser links to has sources that are ancient. I looked up on Chris Kresser, and he is a licenced acupuncturist trained in some version of chinese medicine. This doesn't exactly make me trust him as a source of medical information.

From what I've read, in big-name medical journals you can find that the case about cholesterol is hardly settled for good, but the mainstream medical opinion still is that LDL is very bad for you. Pretty much all mainstream arterial disease risk calculators implement this fact. If I have a patient who has high cholesterol and I don't react, I'm considered a bad doctor by at least 99 % of my colleagues and every single one of my professors. Every textbook I know of can tell you that LDL is bad for you. The only doctor in my country that I know of who recommends raising your LDL is a quack selling overpriced supplements. This is the level of consensus at least in Finland.

How am I supposed to update? Edit: see in the following comments how difficult this is Distrusting my entire field in my country in any circumstance doesn't seem feasible. Understand that I'm not a scientist, and not trained in statistics. I'm a doctor and trained to implement science. I think this is the most feasible division of work given the volume of current knowledge. Currently my time is spent at lectures, at the clinic and reading textbooks, and reading journals doesn't seem like a low hanging fruit most of the time.

Please provide proper sources before you spread any medical information. If your provide the goodies, I'm ready to read. Otherwise as an extortion I will treat my future patients with statins like the rest of the medical community :P

This is an important topic to me, and as I said misinformation is deadly. I hope you can forgive whatever indignation I couldn't eliminate.

A couple of sources that medical people consider reliable: www.cochrane.org, heart disease The Framingham Heart Study

Comment author: Yvain 13 December 2012 11:46:04PM *  12 points [-]

Yes, I understand your indignation and I should not have been as quick to spurt that out without more information. I've tried to justify everything I've said above with data from Framingham and the Cochrane Collaboration, but I hope you'll forgive me if I have to lapse into a few sources from less hallowed publications once in a while.

"Eating cholesterol doesn't cause high blood cholesterol."

Since only citing things from Framingham or Cochrane is a hard constraint to keep I am forced to commit the minor sin of citing a work not published in a peer-reviewed journal and refer you to the Framingham Diet Study, a subpart of the Framingham Heart Study whose methodology was published appropriately but whose results for some reason weren't. A guy who tracked down the results reports on them here and finds that

"With one exception there was no discernible association between reported diet intake and serum cholesterol level in the Framingham Diet Study Group. The one exception was a weak negative association between caloric intake and serum cholesterol level in men. [As to] coronary heart disease–was it related prospectively to diet. No relationship was found."

If you'll allow me to step out of Framingham and Cochrane for a moment I can also link to a review on egg consumption and LDL which found that in most of the population there's no relationship.

"Eating saturated fat probably doesn't cause higher blood cholesterol."

This claim can also be supported by the Framingham data above finding no link between diet and serum cholesterol.

You may also be interested in a Cochrane review that finds no effect reducing dietary fat consumption on cardiovascular events, cardiovascular mortality or total mortality, and finds that changing the type of fat slightly decreases cardiovascular events (look how close that confidence interval gets to 1!), but not cardiovascular mortality or total mortality. And here's a non-Cochrane meta-analysis saying pretty much the same thing.

If those are true, either the fat -> blood cholesterol or the blood cholesterol -> CVD connections have to be flawed; I'd say at this point it's more likely the former.

"High blood LDL probably doesn't cause heart disease."

That was irresponsibly speculative as written; I have now edited it out of the original post lest it cause exactly the confusion you worry about. I am impressed by arguments suggesting that the association between cholesterol and heart disease may be more correlational than causal, or at least super-complicated and modulated by weird factors we don't understand like subtype of LDL, but at this point I can't support that with Framingham or Cochrane and it would be irresponsible to say so too loudly.

But I think it's weird how randomized trials keep finding that a lot of drugs that change cholesterol don't change disease risk. The ENHANCE trial of ezetimibe finds it successfully reduced LDL cholesterol but didn't decrease (and may increase) cardiovascular events. The AIM-HIGH trial of niacin finds it successfully raises HDL cholesterol and lowers LDL cholesterol but didn't decrease cardiovascular events. Fibrates may prevent CVD but have no effect on all-cause mortality. The only lipid-lowering drugs that seem to be consistently awesome are, of course, the statins, but they seem to work equally well in high and low cholesterol populations leading some to think they also have non-cholesterol-related effects.

I also just think LDL is a lousy biomarker. A majority of heart attack patients have LDL levels considered normal; this is especially impressive considering the high results of high cholesterol even in the general population.

But in general you're right and I apologize for asserting this one until I can back it up better.

"High blood cholesterol levels are protective against cancer and the mortality gain here probably outweighs any mortality loss from cardiovascular disease."

Framingham found that mortality increased with increasing cholesterol in people younger than 50. In people older than 50 (ie 90% of heart attack victims) it found no relationship (other sources say low cholesterol led to higher mortality in these age groups, but I can't access the paper to check.)

In other studies, the relationship between cholesterol and noncardiovascular mortality has indeed been clearly protective.The article I linked to tries to explain this as people with hidden catabolic diseases (eg cancer) having decreased cholesterol levels. This has become less plausible as longitudinal studies show low cholesterol long precedes disease; see for example Schatz 2001.

Overall I am not as confident in my claim above as I was when I wrote it. I wrote it right after reading Good Calories, Bad Calories, but before double-checking everything in it to avoid effects of confirmation bias. I continue to think the statements above are mostly true, in the vague and weasel-ish way that medical statements are true (true in most subgroups, if you hold risk factors constant, so on). But I suggest reading Good Calories, Bad Calories and doing some more research and checking for yourself: I would be really interested to know what another rationalist who's interested in medicine thinks of it.

I would suggest (not "recommend", I'm nowhere near high-status or high-information enough to "recommend") that you use a better source of monitoring cardiovascular risk than LDL (ApoA/ApoB ratios seem to work), that you suggest an Obvious Healthy Diet Without Twinkies Or Coca-Cola and leave it at that unless you want to wade into the monster-infested radioactive swamp that is nutrition science, and that you continue prescribing statins where indicated.

Comment author: Yvain 09 December 2012 07:03:45AM *  36 points [-]

Sex education (including non-abstinence) may not work at all, and if it does work it works only in a very weak and limited way.

Eating cholesterol doesn't cause high blood cholesterol. Eating saturated fat probably doesn't cause higher blood cholesterol. High blood cholesterol levels are protective against cancer and the mortality gain here probably outweighs any mortality loss from cardiovascular disease. The entire science of cholesterol is confused and terrible and practically every statement you have ever heard that includes the word "cholesterol" is very likely a lie. (link to a readable blog post with some of this, but you can also find it all in big-name medical journals)

The (good!) effect of drinking alcohol on life expectancy is super strong. Drinking wine a few times a week is correlated with up to four years gain in lifespan (effect mostly found in the middle-aged, might not be such a good idea in young), and people who are smart and understand that correlation isn't always causation have amassed some decent evidence that at least some of this might be causal.

Labeling the amount of calories in food (for example on McDonald's restaurant menus) totally fails to change people's eating behaviors at all, no matter how hard people study it. Article here, credit to SarahC for pointing this out to me.

I hate all of these facts and wish they were not true, which makes me a credible source for them.

2012 Survey Results

80 Post author: Yvain 07 December 2012 09:04PM

Thank you to everyone who took the 2012 Less Wrong Survey (the survey is now closed. Do not try to take it.) Below the cut, this post contains the basic survey results, a few more complicated analyses, and the data available for download so you can explore it further on your own. You may want to compare these to the results of the 2011 Less Wrong Survey.

Part 1: Population

How many of us are there?

The short answer is that I don't know.

The 2011 survey ran 33 days and collected 1090 responses. This year's survey ran 23 days and collected 1195 responses. The average number of new responses during the last week was about five per day, so even if I had kept this survey open as long as the last one I probably wouldn't have gotten more than about 1250 responses. That means at most a 15% year on year growth rate, which is pretty abysmal compared to the 650% growth rate in two years we saw last time.

About half of these responses were from lurkers; over half of the non-lurker remainder had commented but never posted to Main or Discussion. That means there were only about 600 non-lurkers.

But I am skeptical of these numbers. I hang out with some people who are very closely associated with the greater Less Wrong community, and a lot of them didn't know about the survey until I mentioned it to them in person. I know some people who could plausibly be described as focusing their lives around the community who just never took the survey for one reason or another. One lesson of this survey may be that the community is no longer limited to people who check Less Wrong very often, if at all. One friend didn't see the survey because she hangs out on the #lesswrong channel more than the main site. Another mostly just goes to meetups. So I think this represents only a small sample of people who could justly be considered Less Wrongers.

The question of "how quickly is LW growing" is also complicated by the high turnover. Over half the people who took this survey said they hadn't participated in the survey last year. I tried to break this down by combining a few sources of information, and I think our 1200 respondents include 500 people who took last year's survey, 400 people who were around last year but didn't take the survey for some reason, and 300 new people.

As expected, there's lower turnover among regulars than among lurkers. Of people who have posted in Main, about 75% took the survey last year; of people who only lurked, about 75% hadn't.

This view of a very high-turnover community and lots of people not taking the survey is consistent with Vladimir Nesov's data showing http://lesswrong.com/lw/e4j/number_of_members_on_lesswrong/77xz 1390 people who have written at least ten comments. But the survey includes only about 600 people who have at least commented; 800ish of Vladimir's accounts are either gone or didn't take the census.

Part 2: Categorical Data

SEX:
Man: 1057, 89.2%
Woman: 120, 10.1%
Other: 2, 0.2%)
No answer: 6, 0.5%

GENDER:
M (cis): 1021, 86.2%
F (cis): 105, 8.9%
M (trans f->m): 3, 0.3%
F (trans m->f): 16, 1.3%
Other: 29, 2.4%
No answer: 11, 0.9%

ORIENTATION:
Heterosexual: 964, 80.7%
Bisexual: 135, 11.4%
Homosexual: 28, 2.4%
Asexual: 24, 2%
Other: 28, 2.4%
No answer: 14, 1.2%

RELATIONSHIP STYLE:

Prefer monogamous: 639, 53.9%
Prefer polyamorous: 155, 13.1%
Uncertain/no preference: 358, 30.2%
Other: 21, 1.8%
No answer: 12, 1%

NUMBER OF CURRENT PARTNERS:
0: 591, 49.8%
1: 519, 43.8%
2: 34, 2.9%
3: 12, 1%
4: 5, 0.4%
6: 1, 0.1%
7, 1, 0.1% (and this person added "really, not trolling")
Confusing or no answer: 20, 1.8%

RELATIONSHIP STATUS:
Single: 628, 53%
Relationship: 323, 27.3%
Married: 220, 18.6%
No answer: 14, 1.2%

RELATIONSHIP GOALS:
Not looking for more partners: 707, 59.7%
Looking for more partners: 458, 38.6%
No answer: 20, 1.7%

COUNTRY:
USA: 651, 54.9%
UK: 103, 8.7%
Canada: 74, 6.2%
Australia: 59, 5%
Germany: 54, 4.6%
Israel: 15, 1.3%
Finland: 15, 1.3%
Russia: 13, 1.1%
Poland: 12, 1%

These are all the countries with greater than 1% of Less Wrongers, but other, more exotic locales included Kenya, Pakistan, and Iceland, with one user each. You can see the full table here.

This data also allows us to calculate Less Wrongers per capita:


Finland: 1/366,666
Australia: 1/389,830
Canada: 1/472,972
USA: 1/483,870
Israel: 1/533,333
UK: 1/603,883
Germany: 1/1,518,518
Poland: 1/3,166,666
Russia: 1/11,538,462

RACE:
White, non-Hispanic 1003, 84.6%
East Asian: 50, 4.2%
Hispanic 47, 4.0%
Indian Subcontinental 28, 2.4%
Black 8, 0.7%
Middle Eastern 4, 0.3%
Other: 33, 2.8%
No answer: 12, 1%

WORK STATUS:
Student: 476, 40.7%
For-profit work: 364, 30.7%
Self-employed: 95, 8%
Unemployed: 81, 6.8%
Academics (teaching): 54, 4.6%
Government: 46, 3.9%
Non-profit: 44, 3.7%
Independently wealthy: 12, 1%
No answer: 13, 1.1%

PROFESSION:
Computers (practical): 344, 29%
Math: 109, 9.2%
Engineering: 98, 8.3%
Computers (academic): 72, 6.1%
Physics: 66, 5.6%
Finance/Econ: 65, 5.5%
Computers (AI): 39, 3.3%
Philosophy: 36, 3%
Psychology: 25, 2.1%
Business: 23, 1.9%
Art: 22, 1.9%
Law: 21, 1.8%
Neuroscience: 19, 1.6%
Medicine: 15, 1.3%
Other social science: 24, 2%
Other hard science: 20, 1.7%
Other: 123, 10.4%
No answer: 27, 2.3%

DEGREE:
Bachelor's: 438, 37%
High school: 333, 28.1%
Master's: 192, 16.2%
Ph.D: 71, 6%
2-year: 43, 3.6%
MD/JD/professional: 24, 2%
None: 55, 4.6%
Other: 15, 1.3%
No answer: 14, 1.2%

POLITICS:
Liberal: 427, 36%
Libertarian: 359, 30.3%
Socialist: 326, 27.5%
Conservative: 35, 3%
Communist: 8, 0.7%
No answer: 30, 2.5%

You can see the exact definitions given for each of these terms on the survey.

RELIGIOUS VIEWS:
Atheist, not spiritual: 880, 74.3%
Atheist, spiritual: 107, 9.0%
Agnostic: 94, 7.9%
Committed theist: 37, 3.1%
Lukewarm theist: 27, 2.3%
Deist/Pantheist/etc: 23, 1.9%
No answer: 17, 1.4%

FAMILY RELIGIOUS VIEWS:
Lukewarm theist: 392, 33.1%
Committed theist: 307, 25.9%
Atheist, not spiritual: 161, 13.6
Agnostic: 149, 12.6%
Atheist, spiritual: 46, 3.9%
Deist/Pantheist/Etc: 32, 2.7%
Other: 84, 7.1%

RELIGIOUS BACKGROUND:
Other Christian: 517, 43.6%
Catholic: 295, 24.9%
Jewish: 100, 8.4%
Hindu: 21, 1.8%
Traditional Chinese: 17, 1.4%
Mormon: 15, 1.3%
Muslim: 12, 1%

Raw data is available here.

MORAL VIEWS:

Consequentialism: 735, 62%
Virtue Ethics: 166, 14%
Deontology: 50, 4.2%
Other: 214, 18.1%
No answer: 20, 1.7%

NUMBER OF CHILDREN
0: 1044, 88.1%
1: 51, 4.3%
2: 48, 4.1%
3: 19, 1.6%
4: 3, 0.3%
5: 2, 0.2%
6: 1, 0.1%
No answer: 17, 1.4%

WANT MORE CHILDREN?

No: 438, 37%
Maybe: 363, 30.7%
Yes: 366, 30.9%
No answer: 16, 1.4%

LESS WRONG USE:
Lurkers (no account): 407, 34.4%
Lurkers (with account): 138, 11.7%
Posters (comments only): 356, 30.1%
Posters (comments + Discussion only): 164, 13.9%
Posters (including Main): 102, 8.6%

SEQUENCES:
Never knew they existed until this moment: 99, 8.4%
Knew they existed; never looked at them: 23, 1.9%
Read < 25%: 227, 19.2%
Read ~ 25%: 145, 12.3%
Read ~ 50%: 164, 13.9%
Read ~ 75%: 203, 17.2%
Read ~ all: 306, 24.9%
No answer: 16, 1.4%

Dear 8.4% of people: there is this collection of old blog posts called the Sequences. It is by Eliezer, the same guy who wrote Harry Potter and the Methods of Rationality. It is really good! If you read it, you will understand what we're talking about much better!

REFERRALS:
Been here since Overcoming Bias: 265, 22.4%
Referred by a link on another blog: 23.5%
Referred by a friend: 147, 12.4%
Referred by HPMOR: 262, 22.1%
No answer: 35, 3%

BLOG REFERRALS:

Common Sense Atheism: 20 people
Hacker News: 20 people
Reddit: 15 people
Unequally Yoked: 7 people
TV Tropes: 7 people
Marginal Revolution: 6 people
gwern.net: 5 people
RationalWiki: 4 people
Shtetl-Optimized: 4 people
XKCD fora: 3 people
Accelerating Future: 3 people

These are all the sites that referred at least three people in a way that was obvious to disentangle from the raw data. You can see a more complete list, including the long tail, here.

MEETUPS:
Never been to one: 834, 70.5%
Have been to one: 320, 27%
No answer: 29, 2.5%

CATASTROPHE:
Pandemic (bioengineered): 272, 23%
Environmental collapse: 171, 14.5%
Unfriendly AI: 160, 13.5%
Nuclear war: 155, 13.1%
Economic/Political collapse: 137, 11.6%
Pandemic (natural): 99, 8.4%
Nanotech: 49, 4.1%
Asteroid: 43, 3.6%

The wording of this question was "which disaster do you think is most likely to wipe out greater than 90% of humanity before the year 2100?"

CRYONICS STATUS:
No, don't want to: 275, 23.2%
No, still thinking: 472, 39.9%
No, procrastinating: 178, 15%
No, unavailable: 120, 10.1%
Yes, signed up: 44, 3.7%
Never thought about it: 46, 3.9%
No answer: 48, 4.1%

VEGETARIAN:
No: 906, 76.6%
Yes: 147, 12.4%
No answer: 130, 11%

For comparison, 3.2% of US adults are vegetarian.


SPACED REPETITION SYSTEMS
Don't use them: 511, 43.2%
Do use them: 235, 19.9%
Never heard of them: 302, 25.5%

Dear 25.5% of people: spaced repetition systems are nifty, mostly free computer programs that allow you to study and memorize facts more efficiently. See for example http://ankisrs.net/

HPMOR:
Never read it: 219, 18.5%
Started, haven't finished: 190, 16.1%
Read all of it so far: 659, 55.7%

Dear 18.5% of people: Harry Potter and the Methods of Rationality is a Harry Potter fanfic about rational thinking written by Eliezer Yudkowsky (the guy who started this site). It's really good. You can find it at http://www.hpmor.com/.


ALTERNATIVE POLITICS QUESTION:

Progressive: 429, 36.3%
Libertarian: 278, 23.5%
Reactionary: 30, 2.5%
Conservative: 24, 2%
Communist: 22, 1.9%
Other: 156, 13.2%

ALTERNATIVE ALTERNATIVE POLITICS QUESTION:
Left-Libertarian: 102, 8.6%
Progressive: 98, 8.3%
Libertarian: 91, 7.7%
Pragmatist: 85, 7.2%
Social Democrat: 80, 6.8%
Socialist: 66, 5.6%
Anarchist: 50, 4.1%
Futarchist: 29, 2.5%
Moderate: 18, 1.5%
Moldbuggian: 19, 1.6%
Objectivist: 11, 0.9%

These are the only ones that had more than ten people. Other responses notable for their unusualness were Monarchist (5 people), fascist (3 people, plus one who was up for fascism but only if he could be the leader), conservative (9 people), and a bunch of people telling me politics was stupid and I should feel bad for asking the question. You can see the full table here.

CAFFEINE:
Never: 162, 13.7%
Rarely: 237, 20%
At least 1x/week: 207, 17.5
Daily: 448, 37.9
No answer: 129, 10.9%

SMOKING:
Never: 896, 75.7%
Used to: 1-5, 8.9%
Still do: 51, 4.3%
No answer: 131, 11.1%

For comparison, about 28.4% of the US adult population smokes

NICOTINE (OTHER THAN SMOKING):
Never used: 916, 77.4%
Rarely use: 82, 6.9%
>1x/month: 32, 2.7%
Every day: 14, 1.2%
No answer: 139, 11.7%

MODAFINIL:
Never: 76.5%
Rarely: 78, 6.6%
>1x/month: 48, 4.1%
Every day: 9, 0.8%
No answer: 143, 12.1%

TRUE PRISONERS' DILEMMA:
Defect: 341, 28.8%
Cooperate: 316, 26.7%
Not sure: 297, 25.1%
No answer: 229, 19.4%

FREE WILL:
Not confused: 655, 55.4%
Somewhat confused: 296, 25%
Confused: 81, 6.8%
No answer: 151, 12.8%

TORTURE VS. DUST SPECKS
Choose dust specks: 435, 36.8%
Choose torture: 261, 22.1%
Not sure: 225, 19%
Don't understand: 22, 1.9%
No answer: 240, 20.3%

SCHRODINGER EQUATION:
Can't calculate it: 855, 72.3%
Can calculate it: 175, 14.8%
No answer: 153, 12.9%

PRIMARY LANGUAGE:
English: 797, 67.3%
German: 54, 4.5%
French: 13, 1.1%
Finnish: 11, 0.9%
Dutch: 10, 0.9%
Russian: 15, 1.3%
Portuguese: 10, 0.9%

These are all the languages with ten or more speakers, but we also have everything from Marathi to Tibetan. You can see the full table here..

NEWCOMB'S PROBLEM
One-box: 726, 61.4%
Two-box: 78, 6.6%
Not sure: 53, 4.5%
Don't understand: 86, 7.3%
No answer: 240, 20.3%

ENTREPRENEUR:
Don't want to start business: 447, 37.8%
Considering starting business: 334, 28.2%
Planning to start business: 96, 8.1%
Already started business: 112, 9.5%
No answer: 194, 16.4%

ANONYMITY:
Post using real name: 213, 18%
Easy to find real name: 256, 21.6%
Hard to find name, but wouldn't bother me if someone did: 310, 26.2%
Anonymity is very important: 170, 14.4%
No answer: 234, 19.8%

HAVE YOU TAKEN A PREVIOUS LW SURVEY?
No: 559, 47.3%
Yes: 458, 38.7%
No answer: 116, 14%

TROLL TOLL POLICY:
Disapprove: 194, 16.4%
Approve: 178, 15%
Haven't heard of this: 375, 31.7%
No opinion: 249, 21%
No answer: 187, 15.8%

MYERS-BRIGGS
INTJ: 163, 13.8%
INTP: 143, 12.1%
ENTJ: 35, 3%
ENTP: 30, 2.5%
INFP: 26, 2.2%
INFJ: 25. 2.1%
ISTJ: 14, 1.2%
No answer: 715, 60%

This includes all types with greater than 10 people. You can see the full table here.

Part 3: Numerical Data

Except where indicated otherwise, all the numbers below are given in the format:

mean+standard_deviation (25% level, 50% level/median, 75% level) [n = number of data points]

INTELLIGENCE:

IQ (self-reported): 138.7 + 12.7 (130, 138, 145) [n = 382]
SAT (out of 1600): 1485.8 + 105.9 (1439, 1510, 1570) [n = 321]
SAT (out of 2400): 2319.5 + 1433.7 (2155, 2240, 2320)
ACT: 32.7 + 2.3 (31, 33, 34) [n = 207]
IQ (on iqtest.dk): 125.63 + 13.4 (118, 130, 133)   [n = 378]

I am going to harp on these numbers because in the past some people have been pretty quick to ridicule this survey's intelligence numbers as completely useless and impossible and so on.

According to IQ Comparison Site, an SAT score of 1485/1600 corresponds to an IQ of about 144. According to Ivy West, an ACT of 33 corresponds to an SAT of 1470 (and thence to IQ of 143).

So if we consider self-report, SAT, ACT, and iqtest.dk as four measures of IQ, these come out to 139, 144, 143, and 126, respectively.

All of these are pretty close except iqtest.dk. I ran a correlation between all of them and found that self-reported IQ is correlated with SAT scores at the 1% level and iqtest.dk at the 5% level, but SAT scores and IQTest.dk are not correlated with each other.

Of all these, I am least likely to trust iqtest.dk. First, it's a random Internet IQ test. Second, it correlates poorly with the other measures. Third, a lot of people have complained in the comments to the survey post that it exhibits some weird behavior.

But iqtest.dk gave us the lowest number! And even it said the average was 125 to 130! So I suggest that we now have pretty good, pretty believable evidence that the average IQ for this site really is somewhere in the 130s, and that self-reported IQ isn't as terrible a measure as one might think.

AGE:
27.8 + 9.2 (22, 26, 31) [n = 1185]

LESS WRONG USE:
Karma: 1078 + 2939.5 (0, 4.5, 136) [n = 1078]
Months on LW: 26.7 + 20.1 (12, 24, 40) [n = 1070]
Minutes/day on LW: 19.05 + 24.1 (5, 10, 20) [n = 1105]
Wiki views/month: 3.6 + 6.3 (0, 1, 5) [n = 984]
Wiki edits/month: 0.1 + 0.8 (0, 0, 0) [n = 984]

PROBABILITIES:
Many Worlds: 51.6 + 31.2 (25, 55, 80) [n = 1005]
Aliens (universe): 74.2 + 32.6 (50, 90, 99) [n = 1090]
Aliens (galaxy): 42.1 + 38 (5, 33, 80) [n = 1081]
Supernatural: 5.9 + 18.6 (0, 0, 1) [n = 1095]
God: 6 + 18.7 (0, 0, 1) [n = 1098]
Religion: 3.8 + 15.5 (0, 0, 0.8) [n = 1113]
Cryonics: 18.5 + 24.8 (2, 8, 25) [n = 1100]
Antiagathics: 25.1 + 28.6 (1, 10, 35) [n = 1094]
Simulation: 25.1 + 29.7 (1, 10, 50) [n = 1039]
Global warming: 79.1 + 25 (75, 90, 97) [n = 1112]
No catastrophic risk: 71.1 + 25.5 (55, 80, 90) [n = 1095]
Space: 20.1 + 27.5 (1, 5, 30) [n = 953]

CALIBRATION:
Year of Bayes' birth: 1767.5 + 109.1 (1710, 1780, 1830) [n = 1105]
Confidence: 33.6 + 23.6 (20, 30, 50) [n= 1082]

MONEY:
Income/year: 50,913 + 60644.6 (12000, 35000, 74750) [n = 644]
Charity/year: 444.1 + 1152.4 (0, 30, 250) [n = 950]
SIAI/CFAR charity/year: 309.3 + 3921 (0, 0, 0) [n = 961]
Aging charity/year: 13 + 184.9 (0, 0, 0) [n = 953]

TIME USE:
Hours online/week: 42.4 + 30 (21, 40, 59) [n = 944]
Hours reading/week: 30.8 + 19.6 (18, 28, 40) [n = 957]
Hours writing/week: 7.9 + 9.8 (2, 5, 10) [n = 951]

POLITICAL COMPASS:
Left/Right: -2.4 + 4 (-5.5, -3.4, -0.3) [n = 476]
Libertarian/Authoritarian: -5 + 2 (-6.2, -5.2, -4)

BIG 5 PERSONALITY TEST:
Big 5 (O): 60.6 + 25.7 (41, 65, 84) [n = 453]
Big 5 (C): 35.2 + 27.5 (10, 30, 58) [n = 453]
Big 5 (E): 30.3 + 26.7 (7, 22, 48) [n = 454]
Big 5 (A): 41 + 28.3 (17, 38, 63) [n = 453]
Big 5 (N): 36.6 + 29 (11, 27, 60) [n = 449]

These scores are in percentiles, so LWers are more Open, but less Conscientious, Agreeable, Extraverted, and Neurotic than average test-takers. Note that people who take online psychometric tests are probably a pretty skewed category already so this tells us nothing. Also, several people got confusing results on this test or found it different than other tests that they took, and I am pretty unsatisfied with it and don't trust the results.

AUTISM QUOTIENT
AQ: 24.1 + 12.2 (17, 24, 30) [n = 367]

This test says the average control subject got 16.4 and 80% of those diagnosed with autism spectrum disorders get 32+ (which of course doesn't tell us what percent of people above 32 have autism...). If we trust them, most LWers are more autistic than average.

CALIBRATION:

Reverend Thomas Bayes was born in 1701. Survey takers were asked to guess this date within 20 years, so anyone who guessed between 1681 and 1721 was recorded as getting a correct answer. The percent of people who answered correctly is recorded below, stratified by the confidence they gave of having guessed correctly and with the number of people at that confidence level.

0-5: 10% [n = 30]
5-15: 14.8% [n = 183]
15-25: 10.3% [n = 242]
25-35: 10.7% [n = 225]
35-45: 11.2% [n = 98]
45-55: 17% [n = 118]
55-65: 20.1% [n = 62]
65-75: 26.4% [n = 34]
75-85: 36.4% [n = 33]
85-95: 60.2% [n = 20]
95-100: 85.7% [n = 23]

Here's a classic calibration chart. The blue line is perfect calibration. The orange line is you guys. And the yellow line is average calibration from an experiment I did with untrained subjects a few years ago (which of course was based on different questions and so not directly comparable).

The results are atrocious; when Less Wrongers are 50% certain, they only have about a 17% chance of being correct. On this problem, at least, they are as bad or worse at avoiding overconfidence bias as the general population.

My hope was that this was the result of a lot of lurkers who don't know what they're doing stumbling upon the survey and making everyone else look bad, so I ran a second analysis. This one used only the numbers of people who had been in the community at least 2 years and accumulated at least 100 karma; this limited my sample size to about 210 people.

I'm not going to post exact results, because I made some minor mistakes which means they're off by a percentage point or two, but the general trend was that they looked exactly like the results above: atrocious. If there is some core of elites who are less biased than the general population, they are well past the 100 karma point and probably too rare to feel confident even detecting at this kind of a sample size.

I really have no idea what went so wrong.  Last year's results were pretty good - encouraging, even. I wonder if it's just an especially bad question. Bayesian statistics is pretty new; one would expect Bayes to have been born in rather more modern times. It's also possible that I've handled the statistics wrong on this one; I wouldn't mind someone double-checking my work.

Or we could just be really horrible. If we haven't even learned to avoid the one bias that we can measure super well and which is most susceptible to training, what are we even doing here? Some remedial time at PredictionBook might be in order.

HYPOTHESIS TESTING:

I tested a very few of the possible hypothesis that were proposed in the survey design threads.

Are people who understand quantum mechanics are more likely to believe in Many Worlds? We perform a t-test, checking whether one's probability of the MWI being true depends on whether or not one can solve the Schrodinger Equation. People who could solve the equation had on average a 54.3% probability of MWI, compared to 51.3% in those who could not. The p-value is 0.26; there is a 26% probability this occurs by chance. Therefore, we fail to establish that people's probability of MWI varies with understanding of quantum mechanics.

Are there any interesting biological correlates of IQ? We run a correlation between self-reported IQ, height, maternal age, and paternal age. The correlations are in the expected direction but not significant.

Are there differences in the ways men and women interact with the community? I had sort of vaguely gotten the impression that women were proportionally younger, newer to the community, and more likely to be referred via HPMOR. The average age of women on LW is 27.6 compared to 27.7 for men; obviously this difference is not significant. 14% of the people referred via HPMOR were women compared to about 10% of the community at large, but this difference is pretty minor. Women were on average newer to the community - 21 months vs. 39 for men - but to my surprise a t-test was unable to declare this significant. Maybe I'm doing it wrong?

Does the amount of time spent in the community affect one's beliefs in the same way as in previous surveys? I ran some correlations and found that it does. People who have been around longer continue to be more likely to believe in MWI, less likely to believe in aliens in the universe (though not in our galaxy), and less likely to believe in God (though not religion). There was no effect on cryonics this time.

In addition, the classic correlations between different beliefs continue to hold true. There is an obvious cluster of God, religion, and the supernatural. There's also a scifi cluster of cryonics, antiagathics, MWI, aliens, and the Simulation Hypothesis, and catastrophic risk (this also seems to include global warming, for some reason).

Are there any differences between men and women in regards to their belief in these clusters? We run a t-test between men and women. Men and women have about the same probability of God (men: 5.9, women: 6.2, p = .86) and similar results for the rest of the religion cluster, but men have much higher beliefs in for example antiagathics (men 24.3, women: 10.5, p < .001) and the rest of the scifi cluster.

DESCRIPTIONS OF LESS WRONG

Survey users were asked to submit a description of Less Wrong in 140 characters or less. I'm not going to post all of them, but here is a representative sample:

- "Probably the most sensible philosophical resource avaialble."
- "Contains the great Sequences, some of Luke's posts, and very little else."
- "The currently most interesting site I found ont the net."
- "EY cult"
- "How to think correctly, precisely, and efficiently."
- "HN for even bigger nerds."
- "Social skills philosophy and AI theorists on the same site, not noticing each other."
- "Cool place. Any others like it?"
- "How to avoid predictable pitfalls in human psychology, and understand hard things well: The Website."
- "A bunch of people trying to make sense of the wold through their own lens, which happens to be one of calculation and rigor"
- "Nice."
- "A font of brilliant and unconventional wisdom."
- "One of the few sane places on Earth."
- "Robot god apocalypse cult spinoff from Harry Potter."
- "A place to converse with intelligent, reasonably open-minded people."
- "Callahan's Crosstime Saloon"
- "Amazing rational transhumanist calming addicting Super Reddit"
- "Still wrong"
- "A forum for helping to train people to be more rational"
- "A very bright community interested in amateur ethical philosophy, mathematics, and decision theory."
- "Dying. Social games and bullshit now >50% of LW content."
- "The good kind of strange, addictive, so much to read!"
- "Part genuinely useful, part mental masturbation."
- "Mostly very bright and starry-eyed adults who never quite grew out of their science-fiction addiction as adolescents."
- "Less Wrong: Saving the world with MIND POWERS!"
- "Perfectly patternmatches the 'young-people-with-all-the-answers' cliche"
- "Rationalist community dedicated to self-improvement."
- "Sperglord hipsters pretending that being a sperglord hipster is cool." (this person's Autism Quotient was two points higher than LW average, by the way)
- "An interesting perspective and valuable database of mental techniques."
- "A website with kernels of information hidden among aspy nonsense."
- "Exclusive, elitist, interesting, potentially useful, personal depression trigger."
- "A group blog about rationality and related topics. Tends to be overzealous about cryogenics and other pet ideas of Eliezer Yudkowsky."
- "Things to read to make you think better."
- "Excellent rationality. New-age self-help. Worrying groupthink."
- "Not a cult at all."
- "A cult."
- "The new thing for people who would have been Randian Objectivists 30 years ago."
- "Fascinating, well-started, risking bloat and failure modes, best as archive."
- "A fun, insightful discussion of probability theory and cognition."
- "More interesting than useful."
- "The most productive and accessible mind-fuckery on the Internet."
- "A blog for rationality, cognitive bias, futurism, and the Singularity."
- "Robo-Protestants attempting natural theology."
- "Orderly quagmire of tantalizing ideas drawn from disagreeable priors."
- "Analyze everything. And I do mean everything. Including analysis. Especially analysis. And analysis of analysis."
- "Very interesting and sometimes useful."
- "Where people discuss and try to implement ways that humans can make their values, actions, and beliefs more internally consistent."
- "Eliezer Yudkowsky personality cult."
- "It's like the Mormons would be if everyone were an atheist and good at math and didn't abstain from substances."
- "Seems wacky at first, but gradually begins to seem normal."
- "A varied group of people interested in philosophy with high Openness and a methodical yet amateur approach."
- "Less Wrong is where human algorithms go to debug themselves."
- "They're kind of like a cult, but that doesn't make them wrong."
- "A community blog devoted to nerds who think they're smarter than everyone else."
- "90% sane! A new record!"
- "The Sequences are great. LW now slowly degenerating to just another science forum."
- "The meetup groups are where it's at, it seems to me. I reserve judgment till I attend one."
- "All I really know about it is this long survey I took."
- "The royal road of rationality."
- "Technically correct: The best kind of correct!"
- "Full of angry privilege."
- "A sinister instrument of billionaire Peter Thiel."
- "Dangerous apocalypse cult bent on the systematic erasure of traditional values and culture by any means necessary."
- "Often interesting, but I never feel at home."
- "One of the few places I truly feel at home, knowing that there are more people like me."
- "Currently the best internet source of information-dense material regarding cog sci, debiasing, and existential risk."
- "Prolific and erudite writing on practical techniques to enhance the effectiveness of our reason."
- "An embarrassing Internet community formed around some genuinely great blog writings."
- "I bookmarked it a while ago and completely forgot what it is about. I am taking the survey to while away my insomnia."
- "A somewhat intimidating but really interesting website that helps refine rational thinking."
- "A great collection of ways to avoid systematic bias and come to true and useful conclusions."
- "Obnoxious self-serving, foolish trolling dehumanizing pseudointellectualism, aesthetically bankrupt."
- "The cutting edge of human rationality."
- "A purveyor of exceedingly long surveys."

PUBLIC RELEASE

That last commenter was right. This survey had vastly more data than any previous incarnation; although there are many more analyses I would like to run I am pretty exhausted and I know people are anxious for the results. I'm going to let CFAR analyze and report on their questions, but the rest should be a community effort. So I'm releasing the survey to everyone in the hopes of getting more information out of it. If you find something interesting you can either post it in the comments or start a new thread somewhere.

The data I'm providing is the raw data EXCEPT:

- I deleted a few categories that I removed halfway through the survey for various reasons
- I deleted 9 entries that were duplicates of other entries, ie someone pressed 'submit' twice.
- I deleted the timestamp, which would have made people extra-identifiable, and sorted people by their CFAR random number to remove time order information.
- I removed one person whose information all came out as weird symbols.
- I numeralized some of the non-numeric data, especially on the number of months in community question. This is not the version I cleaned up fully, so you will get to experience some of the same pleasure I did working with the rest.
- I deleted 117 people who either didn't answer the privacy question or who asked me to keep them anonymous, leaving 1067 people.

Here it is: Data in .csv format , Data in Excel format

Comment author: Yvain 01 December 2012 08:13:46PM 2 points [-]

A score of 1.32 isn't radically different from the mean CRT scores found for psychology undergraduates (1.5), financial planners (1.76), Florida Circuit Court judges (1.23), Princeton Undergraduates (1.63), and people who happened to be sitting along the Charles River during a July 4th fireworks display (1.53). It is also noticeably lower than the mean CRT scores found for MIT students (2.18) and for attendees to a LessWrong.com meetup group (2.69).

I found this by far the most interesting part of this (very good) post. I am surprised I had to learn it hidden inside a mostly unrelated essay. I would certainly like to hear more about this test.

In response to 2012 Survey Results
Comment author: gwern 29 November 2012 05:53:17PM 10 points [-]

The 2011 survey ran 33 days and collected 1090 responses. This year's survey ran 23 days and collected 1195 responses.

Why did you close it early? That seems entirely unnecessary.

One friend didn't see the survey because she hangs out on the #lesswrong channel more than the main site.

I put a link and exhortation prominently in the #lesswrong topic from the day the survey opened to the day it closed.

M (trans f->m): 3, 0.3% / F (trans m->f): 16, 1.3%

3 vs 16 seems like quite a difference, even allowing for the small sample size. Is this consistent with the larger population?

Prefer polyamorous: 155, 13.1%...NUMBER OF CURRENT PARTNERS:... [>1 partners = 4.5%]

So ~3x more people prefer polyamory than are actually engaged in it...

Referred by HPMOR: 262, 22.1%

Impressive.

gwern.net: 5 people

Woot! And I'm not even trying or linking LW especially often.

(I am also pleased by the nicotine and modafinil results, although you dropped a number in 'Never: 76.5%')

TROLL TOLL POLICY: Disapprove: 194, 16.4% Approve: 178, 15%

So more people are against than for. Not exactly a mandate for its use.

Are people who understand quantum mechanics are more likely to believe in Many Worlds? We perform a t-test, checking whether one's probability of the MWI being true depends on whether or not one can solve the Schrodinger Equation. People who could solve the equation had on average a 54.3% probability of MWI, compared to 51.3% in those who could not. The p-value is 0.26; there is a 26% probability this occurs by chance. Therefore, we fail to establish that people's probability of MWI varies with understanding of quantum mechanics.

Sounds like you did a two-tailed test. shminux's hypothesis, which he has stated several times IIRC, is that people who can solve it will not be taken in by Eliezer's MWI flim-flam, as it were, and would be less likely to accept MWI. So you should've been running a one-tailed t-test to reject the hypothesis that the can-solvers are less MWI'd. The p-value would then be something like 0.13 by symmetry.

In response to comment by gwern on 2012 Survey Results
Comment author: Yvain 30 November 2012 04:29:26AM 1 point [-]

Sounds like you did a two-tailed test. shminux's hypothesis, which he has stated several times IIRC, is that people who can solve it will not be taken in by Eliezer's MWI flim-flam, as it were, and would be less likely to accept MWI. So you should've been running a one-tailed t-test to reject the hypothesis that the can-solvers are less MWI'd. The p-value would then be something like 0.13 by symmetry.

Yes, but I imagined someone like Eliezer might have the hypothesis that the math naturally leads to MWI and rationalists who understood the math would realize that.

In response to 2012 Survey Results
Comment author: shminux 29 November 2012 07:50:13PM *  2 points [-]

Are people who understand quantum mechanics are more likely to believe in Many Worlds? We perform a t-test, checking whether one's probability of the MWI being true depends on whether or not one can solve the Schrodinger Equation. People who could solve the equation had on average a 54.3% probability of MWI, compared to 51.3% in those who could not. The p-value is 0.26; there is a 26% probability this occurs by chance. Therefore, we fail to establish that people's probability of MWI varies with understanding of quantum mechanics.

Just wanted to point out a few fallacies in the above:

  • "can solve the Schrodinger Equation" means nothing or less without specifying the problem you are solving. The two simplest problems taught in a modern physics course, the free particle and a one-dimensional infinite square well are hardly comparable with, say, calculating the MRI parameters.

  • self-reporting "can solve the Schrodinger Equation" does not mean one actually can.

  • even then, "can solve the Schrodinger Equation" does not mean "understand quantum mechanics", as it does not require one to understand measurement and decoherence, which is what motivates MWI in the first place.

  • there are many versions of MWI, from literal ("the Universe split into two or more every time something happens") to Platonic ("Mathematical Universe").

Basically, I hope that you realize that this is a prime example of "garbage in, garbage out". I suppose it's a good thing that there was no correlation, otherwise one might draw some unwarranted conclusions from this.

Comment author: Yvain 30 November 2012 04:25:54AM 3 points [-]

The actual survey specified "can solve the Schrodinger equation for a hydrogen atom". Although it is not exactly synonymous with "understands quantum mechanics", you would expect them to be highly correlated.

In response to 2012 Survey Results
Comment author: Epiphany 30 November 2012 02:07:17AM *  2 points [-]

Problem:

The line: "This includes all types with greater than 10 people. You can see the full table here." links to a gif that is inaccurate, has no key to explain oddities, and is of such poor graphical quality that parts of it are actually unreadable.

It may be that the reason that invalid personality types like "INNJ" are listed is due to typos on the part of the survey participants. If so, then great! But it may also be that the person who constructed this graphic put typos in (I consider this fairly likely due to the fact that the graphical quality is so low that some of it's not readable. For instance, the number of INTPs is so unclear I can't even tell what it says - it looks like 113 but your results in the post claim 143). It isn't obvious why the invalid types are there, so a key or note would be nice.

Also, some of the participants had a good idea: if one of your personality dimension letters changes when taking the test multiple times, you can fill it out with an X. Can we add an instruction for them to do this on the next survey?

Comment author: Yvain 30 November 2012 04:21:12AM 5 points [-]

The graphic was automatically generated by a computer program, so there's no chance that typos were introduced. There's no key to explain oddities because I have no way of knowing the explanation any better than you. When in doubt, blame survey takers being trolls.

But I do apologize for the poor graphic quality.

In response to 2012 Survey Results
Comment author: Cthulhoo 29 November 2012 08:21:20AM *  16 points [-]

Before even reading the full details, I want to congratulate to you for the impressive amount of work. The survey period is possibly my favorite time of the year on lesswrong!

EDIT: The links for the raw csv/xls data at the bottom don't seem to work for me.

Comment author: Yvain 29 November 2012 09:00:26AM 5 points [-]

Thank you. That should be fixed now.

In response to 2012 Survey Results
Comment author: Qiaochu_Yuan 29 November 2012 08:24:25AM *  9 points [-]

When you discuss the calibration results, could you mention that the surveyors were told what constituted a correct answer? I didn't take the survey and it isn't obvious from reading this post. Also, could you include a plug for PredictionBook around there? You've included lots of other helpful plugs.

Comment author: Yvain 29 November 2012 08:30:04AM 5 points [-]

Done.

Comment author: Vaniver 27 November 2012 04:12:22AM *  1 point [-]

Please do not take it.

Is there not a way to shut off the survey?

My speculation: V jbhyq chg yrff guna svir creprag punapr ba yrff guna svir uhaqerq crbcyr gnxvat vg, naq yrff guna svir creprag punapr ba zber guna svir gubhfnaq crbcyr gnxvat vg. (Cbvag rfgvzngr sbe gur rkcrpgrq inyhr: rvtugrra uhaqerq.)

Comment author: Yvain 27 November 2012 04:16:32AM 2 points [-]

Shutting off the survey makes it invisible, which means that people can't go back to see how a question was worded or something.

Comment author: Yvain 27 November 2012 04:01:40AM 8 points [-]

This survey is now closed. I am working on analysis and will have the results in a while. Please do not take it.

Also, you may now begin speculating with 95% confidence intervals on how many people took the survey this year.

Comment author: JoachimSchipper 24 November 2012 11:22:02PM 0 points [-]

It bothers me how many of these comments pick nits ("plowing isn't especially feminine", "you can't unilaterally declare Crocker's Rules") instead of actually engaging with what has been said.

(And those are just women's issues; women are not the only group that sometimes has problems in geek culture, or specifically on Less Wrong.)

Comment author: Yvain 25 November 2012 08:11:00PM *  19 points [-]

It sounds like you are complaining that people are treating arguments as logical constructions that stand or fall based on their own merit, rather than as soldiers for a grand and noble cause which we must endorse lest we betray our own side.

If that's not what you mean, can you clarify your point better?

Comment author: therufs 25 November 2012 06:36:39PM 3 points [-]

I voted "equally offensive".

Framing useful skills as being primarily relevant insofar as they fulfill cultural imperatives that a dependent has probably not yet decided whether or not to comply with is harmful both in terms of denigrating the useful skill and in terms of reinforcing the expectation that the cultural imperative will be fulfilled. Assuming the speaker is someone the dependent believes has their best interests at heart, saying "it will help you" instead of "you need" is just a different way of being manipulative.

In a void, either statement is offensive regardless of the dependent's gender. In actuality, I'd submit that it is somewhat more offensive to suggest cooking and cleaning to a female dependent simply because it does not do anything to encourage the dependent to question what everyone else is telling her, whereas I'd guess that there are plenty of cultural messages deterring males from cooking and cleaning.

Comment author: Yvain 25 November 2012 07:12:02PM *  14 points [-]

Framing useful skills as being primarily relevant insofar as they fulfill cultural imperatives that a dependent has probably not yet decided whether or not to comply with is harmful both in terms of denigrating the useful skill and in terms of reinforcing the expectation that the cultural imperative will be fulfilled. Assuming the speaker is someone the dependent believes has their best interests at heart, saying "it will help you" instead of "you need" is just a different way of being manipulative.

Would you feel the same way about "It would help you to do your math homework so you can graduate high school and get a decent job?" After all, the idea that everyone should graduate high school is a cultural imperative, and some teenagers may not yet have decided whether this is important to them.

Comment author: AlexanderD 24 November 2012 08:28:39AM 6 points [-]

The socialization of children into gender roles of conciliation and confrontation begins very early, as can be seen in a study by Clearfield and Nelson. Accordingly, it is not surprising (and jibes with our common sense) to note that men and women tend to respond to challenges in different ways. I think it's probably too broad to say that men "always" resort to violence "immediately," which seems like a deliberately weak phrasing. Rather, I'd say that men and women find different solutions, because of their different perspectives.

Comment author: Yvain 25 November 2012 02:35:02AM 2 points [-]

Yes, I agree that contingently there is statistically more aggression in men. I don't think that's the point; see my response to Miley.

Comment author: MileyCyrus 24 November 2012 03:37:05AM *  7 points [-]

It's not just a stereotype, it's the (exaggerated) truth. For example, in polls about whether citizens approve of whatever war is happening that decade, men are generally more in favor of the war than women.

EDIT: Changed "not a stereotype" to "not just a stereotype".

Comment author: Yvain 25 November 2012 02:32:19AM *  23 points [-]

I don't think "it's the exaggerated truth" is necessarily an excuse to perpetuate stereotypes.

For example, suppose the writer was a white person who played games with a black dungeon master, who had himself previously played mostly with other black people. One game, the writer tries to solve a problem through negotiation when the DM had planned things so that you were supposed to shoot the bad guys. The writer phrases this not as "The DM had failed to plan for this contingency" but instead as "This is why it's hard to be a white person trying to hang out around black people; they just try to solve every problem by shooting at it and don't accept that we white people might think differently than that."

When someone notices this is perpetuating a stereotype, I don't think it would remove the problem to say "No, seriously, black people are involved in a disproportionate number of shootings", even if this were true. The point isn't that every group is demographically exactly the same, it's that we are trying to avoid creating a climate where we immediately and unreflectingly associate certain groups with the worst characteristics they contingently hold in our current society.

I admit that I am holding this post to a higher standard than I would hold other posts, because it is itself a post about social justice. This might sound like I'm being deliberately annoying and trying to say "gotcha!", but it's not just that.

It's more of a sense of fair play and reciprocity, that the would-be social justice crusader understands that watching your speech to avoid stereotypes is kind of difficult and contrary to usual habits of thought, maybe not the hardest thing in the world, but also not so drop-dead simple that you can immediately assume any failure is due to evil intentions. And so they make a good-faith effort to show that they're going to try to be respectful to your group, even if your group doesn't desperately need the respect. It just makes you feel like they're working with you instead just being someone who yells at you. Like there's a dialogue going on where both sides follow rules when talking to one another, instead of "Shut up and listen why I tell you why you're offensive and how you're going to stop."

I totally admit that as a male I'm not too worried that the stereotype of men as thoughtlessly violent is going to have huge effects on my life, and I'm not seriously offended. But it's like...more like how workers get upset when company executives give themselves huge bonuses, then cut worker pay because the company is under financial pressure. And then say that if the workers worry about the pay cuts they're "not team players". The executives might be right when they say financial pressures necessitate pay cuts for workers. They might even be right that giving themselves large bonuses makes a negligible impact on the company's bottom line. It just seems like a potentially disrespectful gesture.

Comment author: AlexanderD 23 November 2012 11:51:32PM *  4 points [-]

Some of these anecdotes really illustrate the loss suffered when a group is insufficiently diverse. This one in particular struck me as a demonstration of the high value of a range of perspectives:

On this afternoon, our characters are venturing into the countryside and come across two emaciated farmers who tell us their fields are unplowed because dark elves from the forest keep attacking them. “They're going to starve if they don't get a crop in the ground,” I declare. “We've got to plow at least one field.” The boys go along with this plan. ...

“It's rusty too,” intones the Dungeonmaster, “and pieces of it keep breaking off. Look, you're not supposed to be farming. You're supposed to go into the forest and find the dark elves. I don't have anything else about the farmers. The elves are the adventure.” Reluctantly, I give up my agricultural rescue plan and we go into the forest to hack at elves.

All too often, people focus on how gender discrimination is unfair to those who are excluded or minimized, but it's also a loss to the group and its goals as a whole.

Comment author: Yvain 24 November 2012 03:19:06AM 14 points [-]

All too often, people focus on how gender discrimination is unfair to those who are excluded or minimized, but it's also a loss to the group and its goals as a whole.

I don't see how this story has anything to do with gender discrimination, unless it's trying to reinforce some stereotype of "Women can come up with peaceful solutions to problems, but men always resort to violence immediately."

In response to comment by Yvain on Circular Altruism
Comment author: fubarobfusco 20 November 2012 10:55:39PM *  1 point [-]

Except perception doesn't work like that. We can have two qualitatively different perceptions arising from quantities of the same stimulus. We know that irritation and pain use different nerve endings, for instance; and electric shock in different quantities could turn on irritation at a lower threshold than pain. Similarly, a dim colored light is perceived as color on the cone cells, while a very bright light of the same frequency is perceived as brightness on the rod cells. A baby wailing may be perceived as unpleasant; turn it up to jet-engine volume and it will be perceived as painful.

Comment author: Yvain 21 November 2012 05:21:04AM *  1 point [-]

Okay, good point. But if we change the argument slightly to the smallest perceivable amount of pain it's still biting a pretty big bullet to say 3^^^3 of those is worse than 50 years of torture.

(the theory would also imply that an infinite amount of irritation is not as bad as a tiny amount of pain, which doesn't seem to be true)

Comment author: Hul-Gil 01 May 2012 06:22:43PM *  10 points [-]

Well, he didn't actually identify dust mote disutility as zero; he says that dust motes register as zero on his torture scale. He goes on to mention that torture isn't on his dust-mote scale, so he isn't just using "torture scale" as a synonym for "disutility scale"; rather, he is emphasizing that there is more than just a single "(dis)utility scale" involved. I believe his contention is that the events (torture and dust-mote-in-the-eye) are fundamentally different in terms of "how the mind experiences and deals with [them]", such that no amount of dust motes can add up to the experience of torture... even if they (the motes) have a nonzero amount of disutility.

I believe I am making much the same distinction with my separation of disutility into trivial and non-trivial categories, where no amount of trivial disutility across multiple people can sum to the experience of non-trivial disutility. There is a fundamental gap in the scale (or different scales altogether, à la Jones), a difference in how different amounts of disutility work for humans. For a more concrete example of how this might work, suppose I steal one cent each from one billion different people, and Eliezer steals $100,000 from one person. The total amount of money I have stolen is greater than the amount that Eliezer has stolen; yet my victims will probably never even realize their loss, whereas the loss of $100,000 for one individual is significant. A cent does have a nonzero amount of purchasing power, but none of my victims have actually lost the ability to purchase anything; whereas Eliezer's, on the other hand, has lost the ability to purchase many, many things.

I believe utility for humans works in the same manner. Another thought experiment I found helpful is to imagine a certain amount of disutility, x, being experienced by one person. Let's suppose x is "being brutally tortured for a week straight". Call this situation A. Now divide this disutility among people until we have y people all experiencing (1/y)*x disutility - say, a dust speck in the eye each. Call this situation B. If we can add up disutility like Eliezer supposes in the main article, the total amount of disutility in either situation is the same. But now, ask yourself: which situation would you choose to bring about, if you were forced to pick one?

Would you just flip a coin?

I believe few, if any, would choose situation A. This brings me to a final point I've been wanting to make about this article, but have never gotten around to doing. Mr. Yudkowsky often defines rationality as winning - a reasonable definition, I think. But with this dust speck scenario, if we accept Mr. Yudkowsky's reasoning and choose the one-person-being-tortured option, we end up with a situation in which every participant would rather that the other option had been chosen! Certainly the individual being tortured would prefer that, and each potentially dust-specked individual* would gladly agree to experience an instant of dust-speckiness in order to save the former individual.

I don't think this is winning; no one is happier with this situation. Like Eliezer says in reference to Newcomb's problem, if rationality seems to be telling us to go with the choice that results in losing, perhaps we need to take another look at what we're calling rationality.


*Well, assuming a population like our own, not every single individual would agree to experience a dust speck in the eye to save the to-be-tortured individual; but I think it is clear that the vast majority would.

In response to comment by Hul-Gil on Circular Altruism
Comment author: Yvain 20 November 2012 10:43:48PM *  6 points [-]

Thank you for trying to address this problem, as it's important and still bothers me.

But I don't find your idea of two different scales convincing. Consider electric shocks. We start with an imperceptibly low voltage and turn up the dial until the first level at which the victim is able to perceive slight discomfort (let's say one volt). Suppose we survey people and find that a one volt shock is about as unpleasant as a dust speck in the eye, and most people are indifferent between them.

Then we turn the dial up further, and by some level, let's say two hundred volts, the victim is in excruciating pain. We can survey people and find that a two hundred volt shock is equivalent to whatever kind of torture was being used in the original problem.

So one volt is equivalent to a dust speck (and so on the "trivial scale"), but two hundred volts is equivalent to torture (and so on the "nontrivial scale"). But this implies either that triviality exists only in degree (which ruins the entire argument, since enough triviality aggregated equals nontriviality) or that there must be a sharp discontinuity somewhere (eg a 21.32 volt shock is trivial, but a 21.33 volt shock is nontrivial). But the latter is absurd. Therefore there should not be separate trivial and nontrivial utility scales.

Comment author: Yvain 16 November 2012 08:00:29PM 17 points [-]

Ever since reading West Hunter I've been intrigued by the idea of mutational load as an explanation for increasing prevalence of mental disorders. It would mean chance of having these disorders would increase with increasing paternal age, and in fact that's what we find. Since everyone's waiting longer to have babies nowadays, that explains some of the change.

I wonder if increasing paternal age explains even more of the change than that article lets on, just because it's accumulating over generations; it's not just that your dad was older, it's that your grandfather was older and therefore your father started with more mutations and so on. You can't measure that effect just by separating people into autistic and neurotypical and then asking them how old their fathers were.

Comment author: thomblake 16 November 2012 06:59:14PM 19 points [-]

This should probably be an open thread comment.

Note that you've seen "several people" out of hundreds recommend Mencius Moldbug. That is not surprising given that he has debated Robin Hanson, and I believe was linked by Robin several times on Overcoming Bias. I'm not sure how you go from "several people" to "so popular". I don't think there's anything to explain.

Comment author: Yvain 16 November 2012 07:21:22PM *  11 points [-]

On the as-yet-unfinished survey, n gbgny bs fvkgrra people identify themselves as "Moldbuggian" and na nqqvgvbany gjb people as "reactionary". Compare this to other categories; for example nobhg avargl rnpu for "libertarian" and "progressive".

(edit: rot13ed number for people who want to predict it in advance)

Comment author: FAWS 04 November 2012 05:43:35PM 20 points [-]

I wonder whether there are visible conversion effects on the redwood question for native metric users? Estimates slightly on the short side and neatly divisible by three because the quick and dirty meter -> feet conversion is multiplying by three?

Comment author: Yvain 09 November 2012 01:00:20AM 3 points [-]

Seems easy to test for once the results are out: sort people into metric/non-metric groups by the "country" question and compare results.

Comment author: Yvain 04 November 2012 09:54:01PM 2 points [-]

I'm traveling on an unpredictable schedule. If I'm in New York at the time (30% probability) I would love to attend.

Comment author: MBlume 04 November 2012 06:45:55PM 15 points [-]

Is income before or after taxes?

Comment author: Yvain 04 November 2012 09:50:12PM 7 points [-]

Before.

Comment author: [deleted] 04 November 2012 12:22:57AM 38 points [-]

Took it!

Did any one else have trouble copy-pasting the links?

I normally score insanely high on Openness to experience (says she of the massive amounts of really weird hobbies), but for this test, I scored really low on Openness. Must be feeling particularly close-minded today. Weird.

In response to comment by [deleted] on 2012 Less Wrong Census/Survey
Comment author: Yvain 04 November 2012 06:09:32AM 7 points [-]

I also got really low openness on that test. I'm suspicious.

Comment author: jpulgarin 03 November 2012 11:20:37PM 24 points [-]

Do casual sex partners count under the "Number of Current Partners" question?

The instructions tell me that higher numbers are for "polyamorous relationships" which makes it seem like a monogamous person who has multiple casual sex partners should answer 0 for that question.

Comment author: Yvain 03 November 2012 11:36:44PM 2 points [-]

Good question. I'm going with "no".

2012 Less Wrong Census/Survey

65 Yvain 03 November 2012 11:00PM

11/26: The survey is now closed. Please do not take the survey. Your results will not be counted.

It's that time of year again.

If you are reading this post, and have not been sent here by some sort of conspiracy trying to throw off the survey results, then you are the target population for the Less Wrong Census/Survey. Please take it. Doesn't matter if you don't post much. Doesn't matter if you're a lurker. Take the survey.

This year's census contains a "main survey" that should take about ten or fifteen minutes, as well as a bunch of "extra credit questions". You may do the extra credit questions if you want. You may skip all the extra credit questions if you want. They're pretty long and not all of them are very interesting. But it is very important that you not put off doing the survey or not do the survey at all because you're intimidated by the extra credit questions.

The survey will probably remain open for a month or so, but once again do not delay taking the survey just for the sake of the extra credit questions.

Please make things easier for my computer and by extension me by reading all the instructions and by answering any text questions in the most obvious possible way. For example, if it asks you "What language do you speak?" please answer "English" instead of "I speak English" or "It's English" or "English since I live in Canada" or "English (US)" or anything else. This will help me sort responses quickly and easily. Likewise, if a question asks for a number, please answer with a number such as "4", rather than "four".

Okay! Enough nitpicky rules! Time to take the...

2012 Less Wrong Census/Survey

Thanks to everyone who suggested questions and ideas for the 2012 Less Wrong Census Survey. I regret I was unable to take all of your suggestions into account, because some of them were contradictory, others were vague, and others would have required me to provide two dozen answers and a thesis paper worth of explanatory text for every question anyone might conceivably misunderstand. But I did make about twenty changes based on the feedback, and *most* of the suggested questions have found their way into the text.

By ancient tradition, if you take the survey you may comment saying you have done so here, and people will upvote you and you will get karma.

Comment author: RobinZ 02 November 2012 08:41:08AM 2 points [-]

Dammit, please tell me that I didn't just waste all that time answering the 2012 Less Wrong Community Survey - I don't want to have to remember all the answers to those tests community members wanted respondents to take if the form I filled out has been taken down...

Comment author: Yvain 02 November 2012 02:12:24PM 8 points [-]

It'll be back up later today and your answers have been saved. Sorry about this.

Comment author: lukeprog 29 October 2012 08:52:33AM 12 points [-]

I can't actually recall where I've seen someone else say that e.g. "An algebraic proof is a series of steps that you can tell are locally licensed because they maintain balanced weights"

The metaphor of a scale is at least a common teaching tool for algebra: see 1, 2, 3.

Comment author: Yvain 29 October 2012 09:18:49AM 7 points [-]

I was taught algebra with a scale in the sixth grade. We had little weights that said "X" on them and learned that you could add or take away "x" from both sides.

In response to comment by Yvain on [Link] Offense 101
Comment author: Kaj_Sotala 25 October 2012 07:32:38AM 36 points [-]

I'm supportive of this idea, but I wonder if people (including me) who make proposals such as "let's scrap the primary school curriculum and fill it with learning that's actually useful" underestimate the amount of useful things that they've learned in primary school, because they no longer remember the origins of that knowledge and have filed it under "those obvious things that everyone knows".

Comment author: Yvain 28 October 2012 12:11:45AM 2 points [-]

Good point, but I am not sure one way or the other. Compare Sargon of Akkad vs. Julius Caesar. I don't think we spent more time in school on one than the other. I think people know about Caesar because he has permeated into the culture. I'm not sure how many of the impressive interesting things all of us know are because of school versus because of cultural exposure.

In response to comment by Yvain on [Link] Offense 101
Comment author: mytyde 25 October 2012 01:52:23AM *  6 points [-]

(Napoleon didn't invade Russia because of cognitive bias. He'd already defeated Russia several times and "invaded" in 1912 with the object of forcing Russia to keep out of Poland and remain in the Continental System. Logistics killed Le Grand Armee.... Napoleon was actually above average height for his time period... the rumor that Napoleon was short is due to a perhaps-intentional failure to convert French measurement height units into British units of the same name, and so there's no basis for a "Napoleon Complex".)

A more interesting question would be "What cognitive biases through history have led us to think of Napoleon as a short person?"

In response to comment by mytyde on [Link] Offense 101
Comment author: Yvain 28 October 2012 12:07:55AM *  1 point [-]

I was thinking more of overconfidence bias and planning fallacy: "I'll just waltz in here and conquer Moscow in a few months...99% chance it works fine."

Comment author: Yvain 27 October 2012 11:30:33PM *  4 points [-]

I think of them as the Sith to our Jedi.

The Jedi say "We will give you ancient knowledge of vast power. But you must promise to use it only for pure truth-seeking and the good of all mankind. You must not use it to serve your own personal ends, or you will be consumed by it. You cannot possibly imagine what dangerous forces you are playing with!"

The Sith say "Ancient knowledge of vast power?! Awesome! Got to remember to use this to win any fights I get in from now on!"

Despite that I don't dislike them as much as I used to. People were relatively helpful to me when a local troll tried to smear me and even the guy responsible eventually apologized. At the same time I learned almost all of the anti-LW-ism on there is just one guy who really really hates us for some reason.

I do sometimes worry that they sometimes fall victim to the "If other people say this probability is only 1%, I will be even more virtuous than they are if I say it's only 0.0001%" fallacy, but they're probably better than most people and I'd hate to go all narcissism of minor differences on them.

In response to [Link] Offense 101
Comment author: Yvain 25 October 2012 12:04:47AM *  25 points [-]

That would be an amazing class. Even dropping the "offensiveness" billing and just advertising it as a class that would expose you to as many new and unconventional ideas as possible would be pretty neat.

While we're asking for the impossible, I'd kind of like to scrap the entire current primary/secondary school curriculum and replace it entirely with rationality. You'd learn math on the way to being able to use Bayes' Theorem. You'd learn English while writing counter-attitudinal essays. You'd learn history because your assignment is to point out what cognitive biases led Napoleon to make the mistake of invading Russia, and how you would have done better in his shoes. And then you'll play a game of Diplomacy (or Civilization IV, or whatever) to prove it. All exams are calibration tests.

People will complain that it might not give people the same breadth of knowledge. But our current curriculum is entirely about signaling breadth of knowledge. I learned about Sargon of Akkad in sixth grade and I have >90% confidence I'm the only person in the class who remembers his name, and that entirely because I'm the sort of person who would read about people like Sargon anyway outside of class. Once the primary/secondary school system is producing a generation of scholars of Mesopotamian history - or even people who can still speak Spanish five years after their high school Spanish class is over - then they can complain about breadth of knowledge.

But if you optimized the entire school experience for learning how to evaluate information and make good choices, maybe some of that would stick.

Comment author: John_Maxwell_IV 22 October 2012 01:04:02AM *  3 points [-]

I'd like to see finer-grained surveying of what fields people work in. Reading this post (esp. PZ Myers' take on WBE roadmap) made me update in the direction of cryonics and uploading not working, and also made me more worried about information cascades on LW in fields where users have little collective expertise.

(It might also be interesting to have another question along the lines of: how informed/accurate does the stuff you've read on LW regarding your field seem to be? What's something important going on in your field that LW should be discussing? Etc. Dunno if the first question is meaningful for most fields though; I'm just throwing this out there.)

Comment author: Yvain 22 October 2012 05:17:55AM 2 points [-]

I am much more likely to include this in a way that is to your liking (or at all) if you give me exact questions.

Comment author: [deleted] 19 October 2012 11:59:57AM *  4 points [-]

I will probably not be implementing any suggestion that requests a multi-checkbox style question... [examples] The reason is that I haven't been able to figure out how to computer process these effectively; I end out with rows of boxes like "hiking,fishing" or "fishing,boating,climbing" and it's apparently beyond my limited skills to get SPSS to separate these out into separate chunks of information....[examples] Or, teach me a good way to solve this problem using Google Forms and SPSS.

I assume you can put the spreadsheet in excel, yes? Excel is much more powerful than Google docs, and I don't know SPSS.

If so, what you need is to add columns for each option, and put in the relevant version of the line below:

=IF(ISNUMBER(SEARCH("hiking",A1)),"hikes"," ")

This will search the text in box A1 (assume column A is the one with answers like "hiking, swimming, sailing" or whatever), and if it contains the text "hiking" will write "hikes", and if not will write nothing (or "doesn't hike")

If I didn't explain well enough, let me know!

ETA: And if you don't mind just having "TRUE" and "FALSE" as your output (instead of "hikes" and " ") then you can also just use

=ISNUMBER(SEARCH("hiking",A1))

Comment author: Yvain 21 October 2012 10:56:22PM 1 point [-]

Thank you. I'm going to try this, and I might ask you for more help if I can't get it to work on my own.

Comment author: ArisKatsaris 21 October 2012 09:15:41PM 1 point [-]

I am reluctant to change questions that have been on the survey since previous years. For example, Will's suggestion to change the Politics question is good, except that if we did it we would no longer be able to confidently say something like "Less Wrong has gotten more liberal since the last survey".

Then will we be forever saddled with a significantly suboptimal question? Here's a compromise solution: This year have both the old question, unchanged, and also have a new one with the five groupings as suggested: "Reactionary", "Conservative", "Progressive", "Libertarian", "Communist"

Then in following years we can keep just this new question, while having a basis for comparison between the old and the new one.

Comment author: Yvain 21 October 2012 09:55:46PM *  1 point [-]

I am doubtful that switching to your version will get any fewer complaints or be any better than the old version, but on the principle of cheap testing I will include the second question. Please tell me exactly what you want for this new question, eg do you want sentence-length descriptions of each position and if so what are they?

Comment author: [deleted] 19 October 2012 10:45:58AM 3 points [-]
  • What's the point of the Chromosomes question? Once you know someone is a cis male or a trans female, does knowing that they have a Y chromosome tell you that much more?
  • The “White (Hispanic)”--“White (non-Hispanic)” dichotomy is weird to non-Americans, and you may want to add an “Other” answer -- or what are (say) Arabs or Maori supposed to answer?
  • I'd split the Children question into “how many children you have” (write-in) and “do you hope to have more children in the future” (with answers “yes, soon”, “yes, later on in my life”, “no” and “not sure”).
  • If you don't want to add more options to “Political” (e.g. libertarian socialist), please add a “None of the above” answer. (Also, I'd say “most strongly identify or lean towards”.) BTW “liberal” and “libertarian” have other meanings (especially outside America), but that's not a big deal given you give examples.
  • As IIRC was suggested after the last survey, you might link to a non-amateur Internet IQ test, if such a thing exists.
  • Maybe ask both total karma and last-30-day karma?
  • What counts as “intelligent life” in P(Aliens)? I'm assuming that octopus/crow/dolphin/bonobo-level intelligence doesn't count, and that humans right after the Upper Palaeolithic Revolution do.
  • In P(Cryonics) you might want to replace “average” with “randomly chosen”, if you mean the average of the probabilities.
  • “Singularity” is ambiguous -- do you mean P(singularity before the year X|singularity ever) = P(singularity after the year X|singularity ever), or P(singularity before the year X) = P(singularity after the year X or never)?
Comment author: Yvain 19 October 2012 11:18:31AM *  1 point [-]

Chromosomes makes that info easier to process and is useful in case a bunch of people put their gender as "other" or don't understand the gender question.

Comment author: Yvain 19 October 2012 11:14:46AM *  10 points [-]

Current status of these suggestions:

  • I will probably not be implementing any suggestion that requests a multi-checkbox style question, like:

"What activities do you enjoy? Check all that apply"

[] Fishing

[] Boating

[] Hiking

[] Climbing

The reason is that I haven't been able to figure out how to computer process these effectively; I end out with rows of boxes like "hiking,fishing" or "fishing,boating,climbing" and it's apparently beyond my limited skills to get SPSS to separate these out into separate chunks of information. I could do it like this:

Do you enjoy fishing?

[] Y

[] N

Do you enjoy boating?

[] Y

[] N

And so on, but the more options you want, the less happy I am doing this. Or, teach me a good way to solve this problem using Google Forms and SPSS.

  • I am reluctant to change questions that have been on the survey since previous years. For example, Will's suggestion to change the Politics question is good, except that if we did it we would no longer be able to confidently say something like "Less Wrong has gotten more liberal since the last survey". I would rather just include a political compass in the Bonus Questions, plus maybe maybe a more complicated one-word political affiliation question.

  • This is also part of my beef with "other", along with the fact that it's going to mean people who are 99% similar to one option but don't feel it perfectly describes them are instead going to pick something that gives us zero information. I very much agree with Vaniver here. I might or might not add it.

  • I'm balancing ability to totally perfectly capture all answers with ability to let people who just want to take a basic survey do that without answering a thousand mostly-similar questions. So while I understand that it might be theoretically desirable to separate out for example race vs. ethnicity, or country of birth vs. country of residence, or asexual romantic relationships versus sexual romantic relationships, I'm reluctant to bloat any section too much more than it's already bloated - especially the one on sex. I can already see someone like that tabloid reporter from a while back going "And also, the latest Less Wrong survey included 256 questions about your sex life!"

  • Can I get around the ethnicity problem by replacing "White (Non-Hispanic)" with "Latino"? It seems like it should work, but I'm suspicious because none of the US surveys I've encountered have ever done it.

  • Kind of want to avoid beating a dead basilisk.

  • IQ suggestions sound good.

  • ACT suggestion sounds good.

  • Most other bonus question suggestions sound good.

  • Happy to include Big Five test, AQ test, etc in the Unreasonably Long Bonus Questions section.

  • Will fix the Singularity question

  • Will probably fix moral views question to mirror PhilPapers version, even though that screws up past-survey-comparison

  • Will correct all typos

2012 Less Wrong Census Survey: Call For Critiques/Questions

20 Yvain 19 October 2012 01:12AM

The first draft of the 2012 Less Wrong Census/Survey is complete (see 2011 here). I will link it below if you promise not to try to take the survey because it's not done yet and this is just an example!

2012 Less Wrong Census/Survey Draft

I want three things from you.

First, please critique this draft. Tell me if any questions are unclear, misleading, offensive, confusing, or stupid. Tell me if the survey is so unbearably long that you would never possibly take it. Tell me if anything needs to be rephrased.

Second, I am willing to include any question you want in the Super Extra Bonus Questions section, as long as it is not offensive, super-long-and-involved, or really dumb. Please post any questions you want there. Please be specific - not "Ask something about abortion" but give the exact question you want me to ask as well as all answer choices.

Try not to add more than five or so questions per person, unless you're sure yours are really interesting. Please also don't add any questions that aren't very easily sort-able by a computer program like SPSS unless you can commit to sorting the answers yourself.

Third, please suggest a decent, quick, and at least somewhat accurate Internet IQ test I can stick in a new section, Unreasonably Long Bonus Questions.

I will probably post the survey to Main and officially open it for responses sometime early next week.

Comment author: tgb 13 October 2012 02:19:59AM 29 points [-]

It seems clear to me that it is a very bad example. I find that consistently the worst part of Eliezer's non-fiction writing is that he fails to separate contentious claims from writings on unrelated subjects. Moreover, he usually discards the traditional view as ridiculous rather than admitting that its incorrectness is extremely non-obvious. He goes so far in this piece as to give the standard view a straw-man name and to state only the most laughable of its proponents' justifications. This mars an otherwise excellent piece and I am unwilling to recommend this article to those who are not already reading LW.

Comment author: Yvain 13 October 2012 04:30:10AM *  41 points [-]

Yeah, I didn't even mind the topic, but I thought this particular sentence was pretty sketchy:

in the virtue theory of metabolism, lack of exercise actually causes weight gain due to divine punishment for the sin of sloth.

This sounds like a Fully General Mockery of any claim that humans can ever affect outcomes. For example:

in the virtue theory of traffic, drinking alcohol actually causes accidents due to divine punishment for the sin of intemperance

in the virtue theory of conception, unprotected sex actually causes pregnancy due to divine punishment for the sin of lust

And selectively applied Fully General Mockeries seem pretty Dark Artsy.

Comment author: Alicorn 07 October 2012 07:04:31AM *  9 points [-]

Snakes can be described as cold and calculating.

...Why? You can ascribe any adjectives to any nouns, but why would you associate these with snakes in particular? I mean, cold, yes, strictly speaking, unless they have been on a warm rock lately. Or even in the sense that they probably lack humanlike emotions, but is that even meaningful to say about a snake, and even if it is, why would you want to put it in such a negative-affect-laden way?

And calculating? My pet snake is probably way worse at math than the average dog, let alone a bright parrot or five-year-old human. What would she even calculate? She finds warmth appealing and cold unappealing and drinks when she's thirsty and strikes when I dangle a dead mouse in front of her and flinches when surprised and hides under her half-a-log when she's got nothing else to do; this is not rocket science, her brain is about the size of a small raisin and it doesn't need to be any bigger.

Comment author: Yvain 10 October 2012 07:26:23AM 13 points [-]

And calculating?

Sure. Have you never heard of an adder?

Comment author: [deleted] 06 October 2012 09:19:40PM -1 points [-]

Are you tentatively planning on January for the next census? I'm interested in helping, if that's something you need.

In response to comment by [deleted] on Open Thread, August 1-15, 2012
Comment author: Yvain 06 October 2012 10:43:33PM 1 point [-]

I am planning on now, but waiting for someone from CFAR who was going to send me a few questions they wanted included.

In response to Blogs by LWers
Comment author: Yvain 05 October 2012 06:51:18AM *  2 points [-]

James_G's blog seems to have vanished from the Internet, which is too bad as I'm looking for an article from it. Does anyone know of a backup anywhere or know a way to contact JamesG himself?

Comment author: [deleted] 03 October 2012 11:17:49PM 23 points [-]

I liked your comment and have a half-formed metaphor for you to either pick apart or develop:

LW/ rationalist types tend towards hard sciences. This requires more System 2 reasoning. Their fields are like computer programs. Every step makes sense, and is understood.

Humanities tends toward more System 1 pattern recognition. This is more akin to a neural network. Even if you are getting the "right" answer, it is coming out of a black box.

Because the rationalist types can't see the algorithm, they assume it can't be "right".

Thoughts?

In response to comment by [deleted] on The Useful Idea of Truth
Comment author: Yvain 04 October 2012 12:02:00AM 7 points [-]

I like your idea and upvoted the comment, but I don't know enough about neural networks to have a meaningful opinion on it.

Comment author: palladias 03 October 2012 08:36:29PM 1 point [-]

Kushiel's Dart (but not the sequels). The worldbuilding is delightful.

Comment author: Yvain 03 October 2012 11:55:52PM 3 points [-]

How strange! I had the exact opposite experience. By mistake I read Scion first. I then went in order, got to the end, and looped around to Dart. I thought Scion stood on its own extremely well and have been recommending friends start with it; I was almost unable to finish Dart and dismissed it as "Guess she hadn't learned how to write yet".

(My friend who started with Scion on my advice also ended up enjoying them all except Dart, and I suggested to Alicorn that she not start with Dart, she refused my suggestion, and then she disliked Dart and didn't read the others which was my prediction all along.)

Kushiel's Avatar continues to be my favorite of all of them, and the new Naamah series isn't bad either.

Comment author: Eliezer_Yudkowsky 02 October 2012 06:21:18PM 0 points [-]

In particular, "post-utopian" is not a real term so far as I know, and I'm using it as a stand-in for literary terms that do in fact have no meaning. If you think there are none of those, Alan Sokal would like to have a word with you.

Comment author: Yvain 03 October 2012 08:36:06PM 32 points [-]

There's a sense in which a lot of fuzzy claims are meaningless: for example, it would be hard for a computer to evaluate "Socrates is kind" even if the computer could easily evaluate more direct claims like "Socrates is taller than five feet". But "kind" isn't really meaningless; it would just be a lot of work to establish exactly what goes into saying "kind" and exactly where the cutoff point between "kind" and "not so kind" is.

I agree that literary critical terms are fuzzy in the same sense as "kind", but I don't think they're necessarily any more fuzzy. For example, replacing "post-utopian" with its likely inspiration "post-colonial", I don't know much about literature, but I feel pretty okay designating Salman Rushdie as "post-colonial" (since his books very often take place against the backdrop of the issues surrounding British decolonization of India) and J. K. Rowling as "not post-colonial" (since her books don't deal with issues surrounding decolonization at all.)

Likewise, even though "post-utopian" was chosen specifically to be meaningless, I can say with confidence that Sir Thomas More's Utopia was not post-utopian, and I bet most other people will agree with me.

The Sokal Hoax to me was less about totally disproving all literary critical terms, and more about showing that it's really easy to get a paper published that no one understands. People elsewhere in the thread have already given examples of Sokalesque papers in physics, computer science, etc that got published, even though those fields seem pretty meaningful.

Literary criticism does have a bad habit of making strange assertions, but I don't think they hinge on meaningless terms. A good example would be deconstruction of various works to point out the racist or sexist elements within. For example, "It sure is suspicious that Moby Dick is about a white whale, as if Melville believed that only white animals could possibly be individuals with stories of their own."

The claim that Melville was racist when writing Moby Dick seems potentially meaningful - for example, we could go back in time, put him under truth serum, and ask him whether that was intentional. Even if it was wholly unconscious, it still implies that (for example) if we simulate a society without racism, it will be less likely to produce books like Moby Dick, or that if we pick apart Melville's brain we can draw some causal connection between the racism to which he was exposed and the choice to have Moby Dick be white.

However, if I understand correctly literary critics believe these assertions do not hinge on authorial intent; that is, Melville might not have been trying to make Moby Dick a commentary on race relations, but that doesn't mean a paper claiming that Moby Dick is a commentary on race relations should be taken less seriously.

Even this might not be totally meaningless. If an infinite monkey at an infinite typewriter happened to produce Animal Farm, it would still be the case that, by coincidence, it was a great metaphor for Communism. A literary critic (or primatologist) who wrote a paper saying "Hey, Animal Farm can increase our understanding and appreciation of the perils of Communism" wouldn't really be talking nonsense. In fact, I'd go so far as to say that they're (kind of) objectively correct, whereas even someone making the relatively stupid claim about Moby Dick above might still be right that the book can help us think about our assumptions about white people.

If I had to criticize literary criticism, I would have a few vague objections. First, that they inflate terms - instead of saying "Moby Dick vaguely reminds me of racism", they say "Moby Dick is about racism." Second, that even if their terms are not meaningless, their disputes very often are: if one critic says "Moby Dick is about racism" and another critic says "No it isn't", then if what the first one means is "Mobdy Dick vaguely reminds me of racism", then arguing this is a waste of time. My third and most obvious complaint is opportunity costs: to me at least the whole field of talking about how certain things vaguely remind you of other things seems like a waste of resources that could be turned into perfectly good paper clips.

But these seem like very different criticisms than arguing that their terms are literally meaningless. I agree that to students they may be meaningless and they might compensate by guessing the teacher's password, but this happens in every field.

Comment author: shminux 02 October 2012 10:40:04PM 1 point [-]

Well, if the universe actually runs on a computer, then presumably that computer includes data for all stars, not just the ones that are visible to us.

Why should it if its purpose is to simulate reality for humans? What's wrong with a version of The Truman Show?

Comment author: Yvain 02 October 2012 10:43:05PM *  0 points [-]

Because since everything would be a simulation, "all stars" would be identical in meaning with "all stars that are being simulated" and with "all stars for which the computer includes data".

In response to comment by [deleted] on Open Thread, October 1-15, 2012
Comment author: palladias 02 October 2012 03:22:21PM 7 points [-]

I disapprove of this thread on the principle that people should be able to idly speculate on their own blog without being harangued elsewhere.

Is this meant to apply just to LessWrongers? Because it seems kosher to discuss and critique blog posts generally in open threads.

Comment author: Yvain 02 October 2012 10:24:12PM 9 points [-]

On second thought, you make a good point. The problem wasn't Vaniver bringing it up, the problem was me not putting clear muflax-like epistemic state warnings on my blog.

Comment author: Nisan 02 October 2012 03:39:19PM 1 point [-]

Suppose we are not living in a simulation. We are to digitize our universe. Do we make our digitization include stars outside the cosmological horizon? By what principle do we decide?

(I suppose you could be asking us to actually digitize the universe, but we want a principle we can use today.)

Comment author: Yvain 02 October 2012 10:17:49PM *  0 points [-]

Well, if the universe actually runs on a computer, then presumably that computer includes data for all stars, not just the ones that are visible to us.

If the universe doesn't run on a computer, then you have to actually digitize the universe so that your model is identical to the real universe as if it were on a computer, not stop halfway when it gets too hard or physically impossible.

I don't think any of these principles will actually be practical. Even the sense-experience principle isn't useful. It would classify "a particle accelerator the size of the Milky Way would generate evidence of photinos" as meaningful, but no one is going to build a particle accelerator the size of the Milky Way any more than they are going to digitize the universe. The goal is to have a philosophical tool, not a practical plan of action.

Comment author: Salutator 02 October 2012 12:27:41PM 4 points [-]

But that's only useful if you make it circular.

Taking you more strictly at your word than you mean it the program could just return true for the majority belief on empirically non-falsifiable questions. Or it could just return false on all beliefs including your belief that that is illogical. So with the right programs pretty much arbitrary beliefs pass as meaningful.

You actually want it to depend on the state of the universe in the right way, but that's just another way to say it should depend on whether the belief is true.

Comment author: Yvain 02 October 2012 10:14:03PM 2 points [-]

That's a problem with all theories of truth, though. "Elaine is a post-utopian author" is trivially true if you interpret "post-utopian" to mean "whatever professors say is post-utopian", or "a thing that is always true of all authors" or "is made out of mass".

To do this with programs rather than philosophy doesn't make it any worse.

I'm suggesting is that there is a correspondence between meaningful statements and universal computer programs. Obviously this theory doesn't tell you how to match the right statement to the right computer program. If you match the statement "snow is white" to the computer program that is a bunch of random characters, the program will return no result and you'll conclude that "snow is white" is meaningless. But that's just the same problem as the philosopher who refuses to accept any definition of "snow", or who claims that snow is obviously black because "snow" means that liquid fossil fuel you drill for and then turn into gasoline.

If your closest match to "post-utopian" is a program that determines whether professors think someone is post-utopian, then you can either conclude that post-utopian literally means "something people call post-utopian" - which would probably be a weird and nonstandard word use the same way using "snow" to mean "oil" would be nonstandard - or that post-utopianism isn't meaningful.

Comment author: [deleted] 02 October 2012 03:55:03AM *  13 points [-]

I disapprove of this thread on the principle that people should be able to idly speculate on their own blog without being harangued elsewhere.

I disapprove of your use of parables to smuggle in your economic hypotheses, rather than arguing for them competently and clearly.

I disapprove of your commentary, because I agree with wedrifid here:

(Claiming to have) mind read negative beliefs and motives in others then declaring them publicly tends to be frowned upon. Certainly it is frowned upon me.

In response to comment by [deleted] on Open Thread, October 1-15, 2012
Comment author: Yvain 02 October 2012 07:17:15AM *  6 points [-]

I disapprove of this thread on the principle that people should be able to idly speculate on their own blog without being harangued elsewhere.

Thank you :)

EDIT: Actually, see here

Comment author: evand 02 October 2012 12:55:10AM 7 points [-]

It bothers me that no one is applying a reversal test here. The paper even calls out intelligence augmentation as the prime example!

I'm inclined to trust Bostrom's well thought out paper on the matter, but I'd be curious to hear opposing views.

Comment author: Yvain 02 October 2012 07:13:16AM *  5 points [-]

I might endorse a certain very specific reversal test.

If I could choose between the current world except that freethinkers are at a significant disadvantage relative to everyone else, versus a world with a four hour workday but we all had to sleep four hours more per night so we still had the same amount of free time, plus our economy was at the same level as in the 1990s...

...then actually I would choose the current world, because the four hours more sleep per night would also apply on the weekends and so totally disrupt the balance, which I hadn't thought of at all in the original post. So never mind.

Comment author: Nisan 02 October 2012 06:31:35AM 0 points [-]

Only propositions that constrain our sensory experience are meaningful.

If it turns out that the cosmologists are wrong and the universe begins to contract, we will have the opportunity to make contact with the civilization that the colonization starship spawns. The proposition "The starship exists" entails that the probability of the universe contracting and us making contact with the descendants of the passengers of the starship is substantial compared to the probability of the universe contracting.

Comment author: Yvain 02 October 2012 06:55:57AM 4 points [-]

Least convenient possible world - we discover the universe will definitely expand forever. Now what?

Or what about the past? If I tell you an alien living three million years ago threw either a red or a blue ball into the black hole at the center of the galaxy but destroyed all evidence as to which, is there a fact of the matter as to which color ball it was?

Comment author: Nisan 02 October 2012 06:51:20AM 1 point [-]

That doesn't help us decide whether there are stars outside the cosmological horizon.

Comment author: Yvain 02 October 2012 06:52:44AM *  4 points [-]

I feel like writing a more intelligent reply than "Yes it does", so could you explain this further?

Comment author: Eliezer_Yudkowsky 02 October 2012 05:26:28AM 5 points [-]

Koan answers here for:

What rule could restrict our beliefs to just propositions that can be meaningful, without excluding a priori anything that could in principle be true?

Comment author: Yvain 02 October 2012 06:46:20AM *  8 points [-]

If a person with access to the computer simulating whichever universe (or set of universes) a belief is about could in principle write a program that takes as input the current state of the universe (as represented in the computer) and outputs whether the belief is true, then the belief is meaningful.

(if the universe in question does not run on a computer, begin by digitizing your universe, then proceed as above)

Comment author: Vaniver 01 October 2012 10:31:28PM 14 points [-]

So, Yvain posted a blog post recently. I was disappointed. I'm posting about it here because I'll have an easier time following a conversation about my thoughts here than in livejournal comments. I will note that he claims the post is, at most, 60% serious, but that seems at least ten thousand times too high.

A major supporting claim is that if modafinil were legal, it would become expected, and everything would be harder to match the increased ability of humans to be productive.

So the religious people flunk out, everyone else has to work much harder, and in the end no student gains. Arguably future patients might gain from having better trained doctors, but I think this wildly overestimates the usefulness of the medical education system.

A parable:

In the Old Country, the people once did not know of iodine. It was not illegal, but only a very specific kind of geek would eat dried seaweed carried long miles on the backs of beasts and men. One day, a stranger came to the village, preaching of this mysterious substance, claiming that its consumption would make all men cleverer.

The elders convened and discussed this 'boon,' if you could call it that. If one man is cleverer, he profits, but if all men are cleverer, then no man profits. No elder spoke this more loudly than the one whose wife feasted on seaweed, and whose children were free of the stunted look of cretinism. To spare the people from having harder lives, the elders sent this stranger on his way, to not change the ways of the village.

A commentary:

Yvain has seen the misery of Haiti and India firsthand; but it seems only with his eyes.

Comment author: Yvain 02 October 2012 02:16:33AM *  12 points [-]

Yvain has seen the misery of Haiti and India firsthand; but it seems only with his eyes.

I very specifically mentioned potential First World outlays to Third World countries as exceptions to my point. For example, I said:

There may be useful indirect actions, like advancing technology, increasing tax revenue that can be spent on useful absolute goods, and increasing the amount that flows as charity to the Third World (emphasis added)

Other than that, my entire argument was based on the "happiness follows economic growth up to a certain point, then stops" argument that has been mentioned here so many times before. That means a parable talking about how great certain interventions could be for the Third World is irrelevant; the post was very specifically and explicitly aimed at the First.

(I also think the benefits from lack of iodine deficiency are a lot less siphon-away-able)

The "60% serious" number may indeed be too high, though. I meant it to signal that I thought the argument was correct in all of its main points, but probably falls apart because the increase in productivity would produce very small benefits rather than no benefits, and "very small benefits" multiplied by the entire economy still end out pretty huge, especially if some of them end out in the Third World through the indirect methods I mentioned earlier.

Comment author: Yvain 29 September 2012 05:17:00AM 5 points [-]

I've edited this in a way that hopefully removes some of the controversy. Thanks to everyone who voted in the poll here. Actually, wait, no, the opposite of that. The two options ended out perfectly balanced, plus a bunch of people wanted me to make it even snarkier, and it was super confusing.

Anyway, I decided to respect the split poll by making a combination of the two drafts. The name has been changed to "the marginal fallacy", credit to James_G (sorry, Konkvistador, but I really do think that the fallacy of accident is something slightly different), but I kept Worst Argument In The World as a subtitle.

I deleted the euthanasia example, both because it was overkill on the "X is murder" examples and to exactly balance the liberal and conservative examples at three each. Then I heavily edited most of the others, and added to the end a paragraph about how maybe this pattern could be useful in sparking conversation. Then I added some footnotes and just a tiny bit of snark to satisfy the pro-snark contingent.

Hopefully this will be a less than entirely unsatisfactory compromise.

Comment author: khafra 28 September 2012 05:54:07PM 1 point [-]

Yes! I want a pilot program of building a 100ft tall mast for each block in a school district, with a zipline leading down to the school. Imagine the saved time for everyone who no longer has to wait for school busses on their morning commute!

Comment author: Yvain 29 September 2012 12:53:41AM *  6 points [-]

I investigated a similar idea for a conworld once and ended up rejecting it.

AFAICT ziplines with modern technology really aren't good at covering long distances. I didn't study the math, but just eyeballing existing long-distance ziplines it seems you need approximately 1 meter height for 10-20 meters traveled. The average distance to school in the US seems to be between 1 mile and 5 miles depending on who you ask. Let's take the lower option. To go 1 mile you'd need a 250-500 foot mast. But that's just on average; some people will live two miles away and need 500-1000 foot masts - up to as high as the Stratosphere Tower in Vegas.

Not only do you have to pay for a Stratosphere Tower on every block (there are 72000 blocks in Manhattan!), not only do you have to tolerate a forest of huge towers that will probably lower land value, but you've also got to get kids up a 500 foot tower every morning, which means realistically that you're paying for some really good elevators. And we're still only saving kids a two mile bike ride or five minutes waiting for a bus!

You could limit it to the blocks closest to the school to decrease max tower height, but that would also limit the benefit.

Comment author: Yvain 23 September 2012 09:01:35AM *  6 points [-]

One difference I haven't seen mentioned: in American liberal arts colleges, you can pretty much mix and match any combination of courses you want as long as you reach a certain (small) required number of courses in your major (which you don't have to decide until halfway through or so, and can always change with minimal penalty). In England (disclaimer: I've never studied there, but I've studied in Ireland which I think is similar) you get accepted to one course and with a few exceptions you've got to just study that. If you're a dilettante who likes experiencing a little bit of everything, America might be better for you. On the other hand, many American colleges require that you take a certain amount of everything, and if you know what you want and don't want to waste time satisfying a Literature or a Foreign Languages requirement, you might prefer the English system. Obviously I know nothing about Hungary.

If you decide to stay in Hungary and pursue medicine, you might be interested in the local equivalent of an MD/Ph.D program. It would allow you to have the MD and therefore the career prospects, while still being involved in pure research. I think becoming a doctor and then spending the rest of your life in academia is probably at least as easy as getting a Ph.D in biology and then living comfortably in academia (source: wild guess) and you would have the MD to fall back on if the academia didn't work out. The only thing you'd have to worry about is being able to make it through medical school and maybe a few years of regular medical practice while you built up your research credentials enough to switch to full-time academic work. I know nothing about the Hungarian system, but some residencies in the States have specific "research tracks" for people who want to end up in academia.

Comment author: Yvain 22 September 2012 08:13:08PM 16 points [-]

Yet another example of the Sequences plagiarizing the future

Comment author: Thrasymachus 18 September 2012 08:30:55AM *  15 points [-]

One anecdote given the 'PR' worries raised:

I have never read the sequences. After reading Luke's post, I am much less likely to: the impression given is the sequences are generally idiosyncratic takes which recapitulate an already existing and better organized literature. I also think it is more likely the sequences are overrated, either through readers being unaware their (or similar) insights have already been made, or lacking the technical background to critique them.

It also downgraded my estimate of the value of EY's work. Although I was pretty sceptical, I knew there was at least some chance that the sequences really were bursting with new insights and that LW really was streets ahead of mainstream academia. This now seems much less likely - although I don't think EY is a plagiarist, it seems most of the sequences aren't breaking new ground, but summarizing/unwittingly recapitulating insights that have already been made and taken further elsewhere.

So I can see the motivation for EY to defend that their originality: his stock goes down if the sequences are neat summaries but nothing that new rather than bursting with new and important insights, and EY's stock is important for things like donations, public perception of him and the SI, etc. (Both my likelihood of donating and my regard for SI has been lowered a bit by this post and comments). However, EY's way of responding to (weakly implied) criticism with catty arrogance compounds the harm.

Comment author: Yvain 18 September 2012 10:28:23AM *  36 points [-]

If you are at all interested in rationality it would be a huge shame for you to skip the Sequences.

Yes, a lot of the material in the Sequences could also be obtained by reading very very carefully a few hundred impenetrable scholarly books that most people have never heard of in five or ten different disciplines, supplemented by a few journal articles, plus some additional insights by "reading between the lines", plus drawing all the necessary connections between them. But you will not do this.

The Sequences condense all that information, put it in a really fun, really fascinating format, and transfer all of it into the deepest levels of your brain in a way that those hundred books wouldn't. And then there's some really valuable new material. Luke and Eliezer can argue whether the new material is 30% of the Sequences or 60% of the Sequences, but either number is still way more output than most people will produce over their entire lives.

If your worry is that they will just be recapitulating things you already know, I am pretty doubtful; I don't know your exact knowledge level, but they were pretty exciting for me when I first read them and I had college degrees in philosophy and psychology which are pretty much the subjects covered. And if they are new to you, then from a "whether you should read them" point of view it doesn't matter if Eliezer copied them verbatim off Wikipedia.

Seriously. Read the Sequences. Luke, who is the one arguing against their originality above, says that they are the one book he would like to save if there was an apocalypse. I would have to think a long time before saying the same but they're certainly up there.

Also, as a fellow doctor interested in utiltiarianism/efficient charity, I enjoyed your blog and associated links.

Comment author: Eliezer_Yudkowsky 14 September 2012 01:43:22AM 7 points [-]

Keep in mind that it's not "more people" it's more "people who participate in meta threads on Less Wrong". I've observed a tremendous divergence between the latter set, and "what LWers seem to think during real-life conversations" (e.g. July Minicamp private discussions of LW which is where the anti-troll-thread ideas were discussed, asking what people thought about recent changes at Alicorn's most recent dinner party). I'm guessing there's some sort of effect where only people who disagree bother to keep looking at the thread, hence bother to comment.

Some "people" were claiming that we ought to fix things by moderation instead of making code changes, which does seem worth trying; so I've said to Alicorn to open fire with all weapons free, and am trying this myself while code work is indefinitely in progress. I confess I did anticipate that this would also be downvoted even though IIRC the request to do that was upvoted last time, because at this point I've formed the generalization "all moderator actions are downvoted", either because only some people participate in meta threads, and/or the much more horrifying hypothesis "everyone who doesn't like the status quo has already stopped regularly checking LessWrong".

I'm diligently continuing to accept feedback from RL contact and attending carefully to this non-filtered source of impressions and suggestions, but I'm afraid I've pretty much written-off trying to figure out what the community-as-a-whole wants by looking at "the set of people who vigorously participate in meta discussions on LW" because it's so much unlike the reactions I got when ideas for improving LW were being discussed at the July Minicamp, or the distribution of opinions at Alicorn's last dinner party, and I presume that any other unfiltered source of reactions would find this conversation similarly unrepresentative.

Comment author: Yvain 14 September 2012 06:16:20PM *  13 points [-]

I will be starting another Less Wrong Census/Survey in about three weeks; in accordance with the tradition I will first start a thread asking for question ideas. If you can think of a good list of opinions you want polled in the next few weeks, consider posting them there and I'll stick them in.

Comment author: Yvain 11 September 2012 08:31:57PM 3 points [-]

Sex is isomorphic to Eliezer Yudkowsky

Nice.

Comment author: Yvain 11 September 2012 08:58:00PM *  20 points [-]

That which can be destroyed by torture should be.

...I don't like this game anymore :(

Comment author: Yvain 11 September 2012 08:31:57PM 3 points [-]

Sex is isomorphic to Eliezer Yudkowsky

Nice.

Comment author: [deleted] 11 September 2012 08:00:21PM *  6 points [-]

What's the alternative to rationally debating ideas that violate our moral sensibilities, assuming some people hold them?

When inviting a guest speaker for an honorarium to hold forth in front of an audience on a subject that affects few or none of them directly, and just giving them that platform without any semblance of discourse apart from taking questions at the end...yeah, I'm gonna say "Rationally debating ideas that violate our moral sensibilities" is not what was going on. Doubly so since in many cases those ideas actually affirm the moral sensibilities of some fair portion of the population.

Also, I bet (and correct me if I'm wrong) that whatever debate you've watched was not about "Let's round up the [Other Folk] and execute them." My guess is it was either about allowing them voluntary euthanasia, allowing abortion or infanticide on the part of their parents, or ceasing to specifically allocate scarce health resources to them.

Yeah, you're wrong -- we're talking about folks who honestly and straightforwardly suggested it was an ethical good to terminate the lives of these people which they felt either had no value (usually through "gentle" methods of euthanasia, and I do not mean voluntarily applied), or had such small value in comparison to their suffering that it was worth it. This is not hyperbole -- though I find it interesting you found the idea so difficult to believe straight up that your interpretation must be that I'm just flipping my lid over a loose patternmatch and couldn't have possibly understood that right. It suggests you think it doesn't happen often enough for rational people to be concerned about.

That means that what we're really talking about is "Any idea that can be massaged into sounding like an idea that violates our moral sensibilities is 100% certainly wrong, or should never be discussed, or needs more egg-throwing."

No, you're not listening to what I'm actually saying, you're just assuming from the get-go that I'm a screaming mindless chimp flinging feces because The Bad Thing Is Bad.

Comment author: Yvain 11 September 2012 08:22:18PM *  11 points [-]

You're right, I apologize.

(although to be fair, you did say you watched "calm and reasonable people sincerely debate this" and that people were objecting to it being "casually debated", so I don't think it was my fault for assuming it was a debate rather than one person going on about it unopposed.)

Now I'm very curious what exactly was going on, although I understand if you don't want to look like you're pointing fingers at specific people.

Comment author: [deleted] 11 September 2012 06:56:15PM *  5 points [-]

Yeah, no. I've watched perfectly calm and reasonable-sounding people sincerely debate whether some group of other people (queer folks and disabled folks come to mind) have a right to exist that should not be overridden in favor of euthanasia to satisfy their own utility functions. I've watched this happen in the halls of supposedly respectable institutions while members of the group under discussion protested outside.

The people going "Hey, this is really fucked up that our right to exist can just be casually debated with or without us in a mainstream, powerful institution, and our not raising a fuss about that is apparently more important to people than the actual suggestion" weren't polite or unemotional, but they did seem to understand the situation for what it was a whole lot better than the folks inside. I'm not sure I want to be around people who can't perform that kind of sanity check on occasion.

Comment author: Yvain 11 September 2012 07:40:25PM *  3 points [-]

What's the alternative to rationally debating ideas that violate our moral sensibilities, assuming some people hold them?

Is it to declare with 100% certainty that any idea that violates our moral sensibilities is false? Is it to say that maybe there's a small chance that ideas violating our moral sensibilities are true, but even so we must never discuss them so if they're true we're out of luck and will never reach that true belief? Is it to say we may discuss them, but not rationally - that is, we must let the screaming protesters into the debate so that they can throw eggs and mud onto the debaters because that will improve the quality of discourse?

Also, I bet (and correct me if I'm wrong) that whatever debate you've watched was not about "Let's round up the [Other Folk] and execute them." My guess is it was either about allowing them voluntary euthanasia, allowing abortion or infanticide on the part of their parents, or ceasing to specifically allocate scarce health resources to them.

That means that what we're really talking about is "Any idea that can be massaged into sounding like an idea that violates our moral sensibilities is 100% certainly wrong, or should never be discussed, or needs more egg-throwing."

Comment author: Louie 06 September 2012 01:28:38AM 16 points [-]

I preferred the original version that appeared on your private website.

Once you sanitized it for LW by making it more abstract and pedantic, it lost many of the most biting, hilarious asides, that made it a fun and entertaining to read.

Comment author: Yvain 06 September 2012 05:00:48AM 2 points [-]

So...the poll shows +9 support for making it more biased and snarky, +9 support for making it less biased and snarky, and a tepid rejection of leaving it the way it is. Awkward.

Comment author: Yvain 06 September 2012 12:16:52AM 1 point [-]

Poll!

Comment author: Yvain 06 September 2012 12:17:12AM 21 points [-]

Upvote this comment if you prefer the original.

Comment author: Yvain 06 September 2012 12:16:52AM 1 point [-]

Poll!

Comment author: Yvain 06 September 2012 12:17:02AM *  21 points [-]

Upvote this comment if you prefer this version.

Comment author: Yvain 06 September 2012 12:16:52AM 1 point [-]

Poll!

Cleaning up the "Worst Argument" essay

13 Yvain 06 September 2012 12:09AM

There was a lot of controversy over the Worst Argument essay, which surprised me because the basic point seems hard to argue with. I'd like to change it in response to feedback, with the new title "Guilt by Association". Below is the rough draft for the new version, minus a few links and other finishing touches. Please let me know whether you think it is better or worse than the original, and what specific further changes you think that it needs.

 

David Stove once ran a contest to find the Worst Argument In The World, but he awarded the prize to his own entry, and one that shored up his politics to boot. It hardly seems like an objective process.

If he can unilaterally declare a worst argument, then so can I. I declare the Worst Argument In The World to be Guilt By Association: "If we can apply a word to something, we must judge it the same as we judge more prototypical instances of that word."

Well, it sounds dumb when you put it like that. Who even does that, anyway?

Suppose someone wants to build a statue honoring Martin Luther King Jr. for his nonviolent resistance to racism. An opponent of the statue objects: "But Martin Luther King was a criminal!"

Any historian can confirm this is correct. A criminal is technically someone who breaks the law, and King knowingly broke a law against peaceful anti-segregation protest - hence his famous Letter from Birmingham Jail.

But in this case calling Martin Luther King a criminal is Guilt by Association. The archetypal criminal is a mugger or bank robber. He is driven only by greed, preys on the innocent, and weakens the fabric of society. We don't like criminals precisely because we don't like greed and preying on innocents and weakening the fabric of society.

The opponent is saying "Because you don't like criminals, and Martin Luther King is a criminal, you should stop liking Martin Luther King." But King doesn't share any of the features that made us dislike criminals in the first place. Therefore, even though he is a criminal, there is no reason to dislike King. The force of the opponent's argument comes solely from the category "criminal" associating King with people who are bad.  It totally fails to prove King was bad himself.

This all seems so nice and logical when it's presented in this format. Unfortunately, it's also one hundred percent contrary to instinct: the urge is to respond "Martin Luther King? A criminal? No he wasn't! You take that back!" This is why the Worst Argument In The World is so successful. As soon as you do that you've fallen into their trap. Your argument is no longer about whether you should build a statue, it's about whether King was a criminal. Since he was, you have now lost the argument.

Ideally, you should just be able to say "Well, King was the good kind of criminal." But that seems pretty tough as a debating maneuver in real life. Let's look at some political arguments that I think typify Guilt by Association.

I

On my way to work every day, I used to pass a sign reading "ABORTION IS MURDER" The archetypal murder is Charles Manson breaking into your house and shooting you. This sort of murder is bad for at least four reasons: you prefer not to die, you have various thoughts and hopes and dreams that would be snuffed out, your family and friends would be heartbroken, and the rest of society has to live in fear until Manson gets caught. If you define murder as "killing another human being", then abortion is technically murder. But it has none of the downsides of murder Charles Manson style.

If your objection to murder is predicated entirely upon the four reasons above, then abortion might qualify as "murder", but it doesn't share any of the characteristics that make you object to murder of the usual sort. The argument is entirely associative: "Look over there in, that bin marked 'MURDER'. Charles Manson and an abortion doctor! The one is standing suspiciously close to the other, don't you think?"

(some people have tried to solve this problem by defining "murder" as "the unlawful killing of a human being" and then pointing out that abortion is legal. This is exactly as clever as redefining "criminal" to mean "a person who breaks the law but is not Martin Luther King." Cut out the fallacy at its root, not at its branches!)

This argument is relatively clear-cut, but other real world arguments are more complicated.

II

Whenever the airports consider singling out people of Middle Eastern descent for extra security checks, someone is bound to object that "Racial profiling is racist." This is true if we define racism as "discriminating based on a person's race". But why do we have a negative reaction to racism in the first place? Well, the prototypical example of racism is the KKK burning crosses in front of black people's houses. This kind of racism has many obvious problems. It's usually based on scientifically inaccurate generalizations about the moral or intellectual value of different groups. It often leads to violence, hate crimes, verbal abuse, or other traumatic experiences. It keeps whole groups of people from achieving their full potential. And it can be belittling and offensive to the people involved.

Let's stick with these four reasons for our discussion, although obviously there are more. Racial profiling seems to avoid the first three sins of racism, but it definitely hits the fourth. So to object that racial profiling is racist could be either useful or fallacious depending on the intent. If the intent were to remind people that racial profiling, like KKK cross burning, can be belittling and offensive, then that's a worthy goal. If the intent were to stick racial profiling in a bin with Hitler, David Duke, Francis Galton, and the people who bombed the Baptist Church in Birmingham - and then say "Look what company it keeps!", then it's a fallacy. Racial profiling isn't empirically false, isn't violent, and doesn't keep groups of people down. When we hear it called "racist", most of the revulsion we naturally hear at the word should be dismissed as irrelevant to this particular case.

There are a lot of ways to mention that racial profiling is belittling and offensive without using the r-word; for example, you could say "Racial profiling is belittling and offensive." This conveys the one accurate point of "Racial profiling is racist" without the extra baggage. While using the latter sentence would not be provably wrong, one would have to wonder about the motives.

III

In some situations that are even less clear cut, I still think I can see Guilt by Association peeking through.

Consider the common refrain that "Capital punishment is murder". Here most of our objections to Charles Manson really do hold. Capital punishment kills someone who doesn't want to die. It cuts short their hopes and dreams. It disappoints the victim's family and friends. And maybe it does make other people live in fear, either because they've got a hidden criminal record or because they know it has a gruesome history of occasionally killing the falsely accused.

But I still don't think this is a good argument. Mansonesque murder has few if any benefits. Capital punishment arguably has more - some people think it decreases the crime rate, and at the very least it makes crime victims and their families breath a sigh of relief. Do the benefits outweigh the costs? I don't know. But taking something with both costs and benefits and then placing it next to something that only has costs and saying "Look! It's exactly like this thing here!" misses the entire point of the argument. If you want to argue that the costs are worse than the benefits, argue that - don't say "It looks suspiciously like something else that has no benefits at all!"

IV

In an earlier version of this post, some people mistook Guilt by Association for a blanket condemnation of any argument from category membership. I don't think things are quite that bad. Discussing category memberships sometimes insightfully point out double standards. For example, if a racist said we should kill all the Canadians, one might respond: "Canadians are people too!" This would be a challenge for the racist to point out exactly why they believe Canadians lack the features that usually make us think killing people are bad, or what special quality Canadians have that outweighs those features.

Perhaps the best explanation of the difference is that accusing someone of Guilt by Association is an invitation to play Rationalist Taboo. It may be they can taboo the emotionally charged category names and still make the argument. Or it may be that the argument instantly falls flat.

But overall I would recommend avoiding this entire style of discourse. Anything valuable you can do with category memberships you can do in a less sweeping way by just saying what you mean (see the "Racial profiling is belittling and offensive" example above.) And anything you can't do in a less sweeping way probably shouldn't be said at all, with honorable exceptions for people who are consciously making arguments from Schelling fences.

V

Are the following examples of Guilt By Association? Are they of the first, second, or third type? Or are they totally legitimate?

1. Efforts to cure hereditary diseases through genetic engineering are eugenics.

2. Evolutionary psychology is sexist.

3. Euthanasia is murder.

4. Marijuana is a drug.

5. Taxation is theft.

6. Prescription medications are poison.

7. Affirmative action is discriminatory.

8. Someone who had sex with a 16 year old when he was 18 is a sex offender.

9. Radical environmentalism is a religion.

10. Mormonism is a cult.

Comment author: SisterY 05 September 2012 06:10:24PM 6 points [-]

In grade school we learn that "X is like Y" is a simile, and "X is Y" is a metaphor, and that there is some crucial difference between the two. Perhaps there is, but I haven't seen an argument to that effect. Mainly, we call both of these "analogy" or "metaphor."

So the argument for tabooing The Worst Argument in the World is that, since many analogies are unusually powerful and people may not recognize that they're analogies rather than perhaps identities, every analogy is The Worst Argument in the World. Even though many analogies are admittedly productive, the class of argument is tabooed because many of its members are problematic.

Doesn't that make the taboo on The Worst Argument in the World itself a species of The Worst Argument in the World?

Comment author: Yvain 05 September 2012 11:09:39PM 2 points [-]

I'm not trying to taboo everything of the form "X is Y".

Consider an analogy to the argumentum ad hominem fallacy. I think it's correct to dub this a fallacy and say it's not a legitimate move in argument. However, some people are stupid, some people are evil, and it may be correct and proper to mention that they are stupid and evil. It just can't be doing the heavy lifting in an argument. Certainly calling people stupid and evil is useful as a slogan, it's useful for introducing evidence against them, it's even valid in some kinds of arguments (For example, "Bob is stupid, so we probably don't want to let him design the nuclear plant.")

I think Worst Argument in the World is the same way. There are some legitimate uses for statements of the "X is in category Y!" form, but actually doing the heavy lifting in a philosophical argument is not one of them. I'd be pretty happy if people just stopped doing it entirely, but I admit that it's possible (although I think unlikely) to keep using it and always be responsible with it.

Comment author: siodine 02 September 2012 05:03:48PM *  2 points [-]

I think the most critical response to the worst argument in the world is that so many people are misunderstanding it (it was better explained on Yvain's blog where he didn't speak in LessWrongese). However, you are right that it is the logical fallacy of accident (as it is probably a form or child or parent of various other types of fallacies), but it's been put in LessWrong's clothes like Yudkowsky has done with other existing biases and fallacies, as such it assumes the LWian worldview and thus imports some nuances which kilobug partly noted.

To your second point, no line is ever drawn on what thing inside cluster-space is outside of the cluster for a given argument. Instead, the entire cluster is banished. Instead, you must argue for the tautology of which the cluster represents (e.g., murder cluster = tautologically bad), and even that's assuming the cluster should be noncontinuous tautologies (shouldn't things farther away from the center of the murder cluster be less bad?). This is no different than the philosophical process of unpacking statements to avoid begging the question.

Comment author: Yvain 05 September 2012 07:03:15AM 1 point [-]

Can you explain what part on my blog you thought was better, so I can maybe replace it here?

Comment author: Yvain 05 September 2012 07:00:18AM *  14 points [-]

I want to respond to James G's critique of this post. First because it was pretty intense, second because I usually enjoy reading his blog, and third because maybe other people have the same objection. I'm doing it here because his blog is closed to comments.

There is no basis to allege that everyone who says, “affirmative action is racist” is trying to position “affirmative action” in the very heart of the “racism” cluster. Clusters-in-thingspace, especially nebulous ones like “racism”, are huge volumes. That affirmative action belongs somewhere in this volume, rather than well outside, is a claim worth making even if affirmative action isn’t a central member...Affirmative action is racist!” draws attention to a cartographic error. “Affirmative action” shouldn’t be remote from “racism”; it is a marginal member of the racism cluster.

I would ask James why exactly we're trying to create a "racism" cluster to begin with. Are we ontologists who place things in categories for fun in our spare time? If so, his cartographic metaphor is apt; we're just trying to draw a map of conceptspace and we should be politely reminded that "affirmative action" is in the wrong part.

But in fact, our real reason for drawing a "racism" cluster is to make hidden inferences (see the section titled "Hidden Inferences" here). Most people see Hitler, the KKK, and South Africa, and decide racism is bad. Therefore, anything in the "racism" cluster is bad. Therefore, most people who want to draw maps of racism are not disinterested cartographers but people trying to convince others that something is bad because it is in that cluster.

To give an example, suppose a mapinguary is trying to lure you into the Amazon jungle to eat you. "Come to the city of Cayenne", it says. "It's in France, so you'll be perfectly safe." You reflect that France is in fact way up in Europe and not covered in jungle at all. So you go to Cayenne and it turns out you're in French Guyana and the mapinguary eats you.

The advice that you should place Cayenne in France is useful for cartographers but dangerous for people who don't want to be eaten by mapinguaries. While I agree with the philosophical point that not all French territories must agree with our archetypal example of France and that Cayenne is a perfectly acceptable marginal example of Frenchness, the practical point is that on the only criterion we're interested in, safety from mapinguaries, Cayenne defies all the inferences we could be expected to make from its Frenchness. Furthermore, the whole reason it was brought up was the hope you would make these bad inferences. It is not always bad and irrational to note that Cayenne is in France, but this particular example was, and there are some things that are such marginal category members that one can in practice be pretty sure the reference is motivated.

I think I agree with you that sometimes "affirmative action is racism" can be useful as a slogan if you believe many of the same features that make the KKK bad also make affirmative action bad; I guess it would be sort of an opener for the discussion "List the reasons you don't like the KKK; now look and you'll see that many of those same things are true of affirmative action." This seems legitimate to me if it's true.

On the other hand, I think there are a lot of people who hate the KKK for reasons that don't apply at all to affirmative action, yet who might still feel they have to dislike affirmative action merely because it's in the category "racism". Trying to trick them into this is the Worst Argument in the World.

This is kind of tricky since in a lot of cases maybe there are ten reasons someone dislikes the KKK, and affirmative action shares only one of these reasons. So it's not completely dishonest, as it may be an honest attempt to point out the one feature both genuinely share. But it's hardly advisable either, especially if we don't expect the person to be able to keep in working memory that nine of the ten reasons they dislike racism don't apply to affirmative action.

I want to eventually retitle this "Guilt by Association Fallacy" (or something) and rewrite it a bit (with that eighth anti-liberal example I promised sewing-machine) but I'll wait for all the criticism to come in, especially Konkvistador's.

Comment author: Yvain 01 September 2012 02:20:44PM 56 points [-]

Do unto others 20% better than you expect them to do unto you, to correct for subjective error.

-- Linus Pauling

Comment author: SisterY 30 August 2012 06:28:30PM 7 points [-]

Again, the objection seems to be more about the particular USE of the argument than the nature of the argument itself (what I call above "non-responsiveness"). I would genuinely like to understand why analogies of the kind you call the Worst Argument in the World are so harmful (and I appreciate your engaging on it). Is it your claim that people are particularly likely to take analogies seriously as arguments, more than other arguments? Is it their very power that makes them so bad?

Rhyming and other poetic tricks, like showing a picture, make statements feel more true to hearers; are those tricks less dangerous than analogy because we (think we) are immune to them? I can kind of intuitively understand what you mean by something being a real argument or not ("as an argument"), but I'm not sure why things taken seriously as arguments are more dangerous than sneaky, non-argument cues that make things seem true.

I wonder if what you really want to destroy are "things effectively masquerading as arguments that aren't really arguments." That class is not exhausted by inexact analogies (which is to say all analogies), nor are all inexact analogies members of that class.

I think metonymy (association, like eugenics --> Hitler) is a much more harmful cognitive sin than metaphor (which at least requires a theory of why things are similar).

Comment author: Yvain 01 September 2012 04:21:45AM 1 point [-]

I wonder if what you really want to destroy are "things effectively masquerading as arguments that aren't really arguments." That class is not exhausted by inexact analogies (which is to say all analogies), nor are all inexact analogies members of that class.

This sounds like a fair summary. I stick to my assertion that what you're calling analogies (and which I would specify are analogies that are not phrased in analogy form and which the overwhelming majority of people never recognize as analogies) are more common and more convincing than most other members of this class.

Comment author: Yvain 27 August 2012 09:38:42PM *  28 points [-]

The challenge is an interesting exercise, and I will try to think up some examples, but your comment also contains an implied accusation which I'd like to respond to first.

By my count, this post includes critiques of four weak right-wing arguments (abortion, euthanasia, taxation, affirmative action) and three weak left-wing arguments (eugenics, sexism, capital punishment). As far as I know, neither side thinks MLK was a criminal. That means I'm 4-3, ie as balanced as it's mathematically possible to get while seven remains an odd number.

And I think the responses I see below justify my choice of examples. Shminux says the pro-choice converse of "abortion is murder" would be "forced pregnancy is slavery"; TGM suggests below it "denying euthanasia is torture". These would be excellent examples of TWAITW if anyone ever asserted them which as far as I know no one ever has. Meanwhile, I continue to walk past signs saying "Abortion Is Murder!" on my way to work every day. I don't know who exactly it would be helping to give "Forced Pregnancy Is Slavery" equal billing with "Abortion Is Murder" here and let my readers conclude that I'm arguing against some fringe position irrelevant to the real world.

If you can think of left-wing WAITWs that are as well-known and catchy as "abortion is murder!", I will happily edit the post to include them (well, to include one of them; otherwise it'll be 5-4 and the leftists will start complaining). The best I can do at the moment is anti-war arguments that seem to equate for example humanitarian intervention in Rwanda with invading your next-door neighbor to steal their land because they're both "war", but that one doesn't come in convenient slogan form as far as I know.

Comment author: Yvain 30 August 2012 02:52:48AM *  3 points [-]

So this may be more complicated than I thought, in that all of the examples below seem really bad to me, but that might just be an example of my personal bias. I think if any of them get, let's say, more than ten upvotes I'll assume they're generally agreed to be a good argument and I'll put them in - does that sound like a reasonable bar? That means upvote them if you think they're worthy of inclusion.

I was trying to think of further liberal examples, and I think some references to "human rights" might qualify - for example, "health care is a human right". The meaning of "human right" that allows us to assert this seems very poorly defined, whereas the meaning of "human right" that allows us to say that negative rights like free speech are human rights seems well-defined, even though I don't agree with it. So calling health care (or housing, or something) a "human right" might be a way of trying to claim that we should view health care as exactly like free speech, free religion, etc, even though it is quite different in that it requires positive action by other people.

I'm not quite willing to include that one just because the total ambiguity in the definition of "human right" makes it pretty hard to pin down exactly how the argument is being made.

EDIT: Just saw "Property is theft" has 15 upvotes. Do people think this one should be added?

Comment author: SisterY 29 August 2012 04:28:48AM 5 points [-]

From that example, it sounds like mindless repetition (non-responsiveness) is the worst argument in the world, whether or not it contains an analogy. What is the special harm of analogy that makes it worse than other kinds of mindless repetition? (Worse than, say, other kinds of seductive, poetic language like rhyming words, a la "if it doesn't fit you must acquit.") And is an analogy still "the worst argument in the world" if it's NOT mindlessly repeated?

Comment author: Yvain 30 August 2012 02:48:19AM 8 points [-]

I don't think it's precisely about mindless repetition. For example:

A: I think eating meat is morally okay, because animals have simple nervous systems and can't feel pain.

B: But meat is murder!

Here even though A spoke first and there is no repetition involved, I still think B's response is inadequate, because B is accusing A of double standards after A has explained the double standard away. The reason why this is more dangerous than (if not worse than) "If the glove won't fit, you must acquit" is that B looks like she is making a novel and nontrivial point and it's not immediately obvious that this is a non-argument already addressed by A's statement (whereas hopefully no one takes the glove argument seriously as an argument)

Comment author: Larks 28 August 2012 01:08:54AM 5 points [-]

Absent any reason to do so, disliking instances of harm simply because they are instances of harm makes no more sense than disliking all elephants simply because they are elephants.

I don't want to assume any metaethical baggage here, but I'm not sure why "because it is an instance of harm" is an acceptable answer but "because it is an instance of theft" is not.

Comment author: Yvain 28 August 2012 01:13:16AM 13 points [-]

Keeping your principle of ignoring meta-ethical baggage, dis-valuing harm only requires one first principle, whereas dis-valuing murder, theft, elephants, etc require an independent (and apparently arbitrary) decision at each concept. Further, it's very suspicious that this supposedly arbitrary decision almost always picks out actions that are often harmful when there are so very many things one could arbitrarily decide to dislike.

Comment author: Larks 27 August 2012 11:48:27PM 7 points [-]

I think this is because of the way you're deconstructing the arguments. In each case, the features you identify which supposedly make us dislike the arcetypal cases are harm-based features. Someone who believed in sanctity instead might identify the category as a value in itself. Attempts to ascribe utilitarian-style values to them, which they supposedly miss the local inapplicability of, risks ignoring what they actually value.

If people genuinely do think murder is wrong simply because it is murder, rather than because it causes harm, then this is not a bad argument.

Comment author: Yvain 28 August 2012 12:32:47AM *  6 points [-]

Absent any reason to do so, disliking all murders simply because they are murders makes no more sense than disliking all elephants simply because they are elephants. You can choose to do so without being logically inconsistent, but it seems like a weird choice to make for no reason. Did you just arbitrarily choose "murder" as a category worthy of dislike, whether or not it causes harm?

At the risk of committing the genetic fallacy, I would be very surprised if their choice of murder as a thing they dislike for its own sake (rather than, say, elephants) had nothing to do with murder being harmful. And although right now I am simply asserting this rather than arguing it, I think it's likely that even if they think they have a deductive proof for why murder is wrong regardless of harm, they started by unconsciously making the WAITW and then rationalizing it.

But I agree that if they do think they have this deductive proof, screaming "Worst argument in the world!" at them is useless and counterproductive; at that point you address the proof.

Comment author: gjm 27 August 2012 09:45:55PM 7 points [-]

I don't understand how you get from "policy debates should not appear one-sided" to "there should be no shortage of weak arguments 'on your side'". Especially if you replace the latter with "there should be no shortage of weak arguments of this sort on your side" -- which is necessary for the challenge to be appropriate -- since there could be correlations between a person's political position and which sorts of fallacies are most likely to infect their thinking.

Comment author: Yvain 27 August 2012 11:35:26PM 9 points [-]

In particular, I predict WAITW use to be correlated with explicit endorsement of sanctity-based rather than harm-based moral values, and we've recently been talking about how that might differ between political groups.

Comment author: SisterY 27 August 2012 07:07:14PM 15 points [-]

Since we think largely in words, pointing out similarities between Thing We Think Is Bad and Thing We Think Is Good requires us to examine the connotations of the words we use. We should be doing that all the time. Just as this alleged "worst argument in the world" can be used to sneak in connotations, it can also be used to force examination of connotations that have previously been sneaked in.

Comment author: Yvain 27 August 2012 11:33:49PM 20 points [-]

I agree. I'm not saying that this form can't be used as a means of examining our intuitions. For example, "meat is murder" is a snappier way of asking "Why, given that we're so worried about harming humans, are we so callous about harming animals?"

But then once the other person answers you with something like "It's because animals have no natural rights" or "Because animals don't have sophisticated enough nervous systems to suffer" or whatever it is they say, the debate has to shift to whether or not that objection is valid. So "but meat is murder!" shouldn't be used as a counterargument to "Animals don't have sophisticated enough nervous systems to suffer", because this latter statement is already answering the question the former was intended to ask.

Comment author: MixedNuts 27 August 2012 09:44:14PM 8 points [-]

seven remains an even number

Either this is a joke or you mean "odd".

Comment author: Yvain 27 August 2012 09:49:52PM 17 points [-]

You saw nothing!

Comment author: Emile 27 August 2012 11:40:03AM 14 points [-]

And okay, a tiny fraction of the time people are just trying to use words as a Schelling fence.

I'm not sure it's that tiny, especially once you're using the "steel man" version of the arguments; i.e. things like "Schelling fences" do not often appear in the reasons given for the disagreement, but that can still be what it boils down to.

People who object to abortion may be objecting to a weakening of the social stigma against the murder of innocents - that social stigma performs a useful function in society, so allowing anything that could be described as "murder of innocents" is perceived as bad, regardless of whether that thing is in itself bad.

In other words, even if words are hidden inferences with leaky generalizations etc. - social norms are still defined in terms of words, and so "pointless" debates over definitions still have their place in discussions of morality. Questions that shouldn't be morally relevant ("is abortion murder?") become so because of the instrumental value of social norms.

So yes, sometimes pulling out a dictionary in the middle of a moral argument may be justified. The discussion can then turn to something more useful, like "is it worse if the norm against murder is slightly weakened, or if women have to keep children they don't want?".

Comment author: Yvain 27 August 2012 09:47:56PM 10 points [-]

Even if that is true (and I stick to my guess that it's only a tiny fraction of the time) I still think deconstructing the argument is valuable. If people's true rejection of abortion is Schelling fences, then let's talk Schelling fences! I would ask why birth doesn't also work as a Schelling fence, and I would get to hear their response, and maybe one of us would change our mind.

But if their true rejection is based on Schelling fences, and instead they're just saying that abortion is murder, there's not much we can do except play Dueling Dictionaries. And the reason that has no chance of working ("Really? Merriam-Webster defines murder as killing a human after birth? Guess I'll go NARAL!") is directly related to it not being their real issue.

Comment author: [deleted] 27 August 2012 04:23:17PM 7 points [-]

That is the author's stated intention. But what he's created is an easily referenceable refutation of weak versions of strong arguments. You're concerned about someone crying "murder" or "theft" instead of thinking. I'm concerned about someone linking to this very popular article instead of thinking, or anyway instead of grappling with a strong argument.

Comment author: Yvain 27 August 2012 09:43:15PM 9 points [-]

As Alex Mennen brought up on my blog, the problem you're worrying about is that someone will say "That's an example of the Worst Argument In The World, which is typically a weak argument", even though that particular version of the argument is actually quite strong.

Luckily, I hear there's a new post addressing exactly that error!

Comment author: RobertLumley 27 August 2012 07:47:42PM 0 points [-]

This. The lack of examples from the left makes me uncomfortable sharing this article with people that will likely see it as an attack on their ideology. If they have some things to cheer for too, they are far more likely to accept it as a good post.

Comment author: Yvain 27 August 2012 09:39:11PM 3 points [-]

See above. Come up with a good leftist example beyond the three already there and I'll add it.

Comment author: [deleted] 27 August 2012 04:37:42PM 32 points [-]

Yvain, here is a challenge. Many of your examples are weak versions of strong right-wing arguments that you do not accept. (by your remark about Schelling fences, it seems you're aware of this). I challenge you to replace each of these examples with a weak version of a strong left-wing argument that you do accept. Since policy debates should not appear one-sided, there should be no shortage of weak arguments "on your side." And it would be an interesting kind of ideological Turing test.

Perhaps I'm wrong about "what side you're on" and you already accept the strong right-wing arguments. In which case you got me, well done!

Comment author: Yvain 27 August 2012 09:38:42PM *  28 points [-]

The challenge is an interesting exercise, and I will try to think up some examples, but your comment also contains an implied accusation which I'd like to respond to first.

By my count, this post includes critiques of four weak right-wing arguments (abortion, euthanasia, taxation, affirmative action) and three weak left-wing arguments (eugenics, sexism, capital punishment). As far as I know, neither side thinks MLK was a criminal. That means I'm 4-3, ie as balanced as it's mathematically possible to get while seven remains an odd number.

And I think the responses I see below justify my choice of examples. Shminux says the pro-choice converse of "abortion is murder" would be "forced pregnancy is slavery"; TGM suggests below it "denying euthanasia is torture". These would be excellent examples of TWAITW if anyone ever asserted them which as far as I know no one ever has. Meanwhile, I continue to walk past signs saying "Abortion Is Murder!" on my way to work every day. I don't know who exactly it would be helping to give "Forced Pregnancy Is Slavery" equal billing with "Abortion Is Murder" here and let my readers conclude that I'm arguing against some fringe position irrelevant to the real world.

If you can think of left-wing WAITWs that are as well-known and catchy as "abortion is murder!", I will happily edit the post to include them (well, to include one of them; otherwise it'll be 5-4 and the leftists will start complaining). The best I can do at the moment is anti-war arguments that seem to equate for example humanitarian intervention in Rwanda with invading your next-door neighbor to steal their land because they're both "war", but that one doesn't come in convenient slogan form as far as I know.

The noncentral fallacy - the worst argument in the world?

157 Post author: Yvain 27 August 2012 03:36AM

Related to: Leaky Generalizations, Replace the Symbol With The Substance, Sneaking In Connotations

David Stove once ran a contest to find the Worst Argument In The World, but he awarded the prize to his own entry, and one that shored up his politics to boot. It hardly seems like an objective process.

If he can unilaterally declare a Worst Argument, then so can I. I declare the Worst Argument In The World to be this: "X is in a category whose archetypal member gives us a certain emotional reaction. Therefore, we should apply that emotional reaction to X, even though it is not a central category member."

Call it the Noncentral Fallacy. It sounds dumb when you put it like that. Who even does that, anyway?

It sounds dumb only because we are talking soberly of categories and features. As soon as the argument gets framed in terms of words, it becomes so powerful that somewhere between many and most of the bad arguments in politics, philosophy and culture take some form of the noncentral fallacy. Before we get to those, let's look at a simpler example.

Suppose someone wants to build a statue honoring Martin Luther King Jr. for his nonviolent resistance to racism. An opponent of the statue objects: "But Martin Luther King was a criminal!"

Any historian can confirm this is correct. A criminal is technically someone who breaks the law, and King knowingly broke a law against peaceful anti-segregation protest - hence his famous Letter from Birmingham Jail.

But in this case calling Martin Luther King a criminal is the noncentral. The archetypal criminal is a mugger or bank robber. He is driven only by greed, preys on the innocent, and weakens the fabric of society. Since we don't like these things, calling someone a "criminal" naturally lowers our opinion of them.

The opponent is saying "Because you don't like criminals, and Martin Luther King is a criminal, you should stop liking Martin Luther King." But King doesn't share the important criminal features of being driven by greed, preying on the innocent, or weakening the fabric of society that made us dislike criminals in the first place. Therefore, even though he is a criminal, there is no reason to dislike King.

This all seems so nice and logical when it's presented in this format. Unfortunately, it's also one hundred percent contrary to instinct: the urge is to respond "Martin Luther King? A criminal? No he wasn't! You take that back!" This is why the noncentral is so successful. As soon as you do that you've fallen into their trap. Your argument is no longer about whether you should build a statue, it's about whether King was a criminal. Since he was, you have now lost the argument.

Ideally, you should just be able to say "Well, King was the good kind of criminal." But that seems pretty tough as a debating maneuver, and it may be even harder in some of the cases where the noncentral Fallacy is commonly used.


Now I want to list some of these cases. Many will be political1, for which I apologize, but it's hard to separate out a bad argument from its specific instantiations. None of these examples are meant to imply that the position they support is wrong (and in fact I myself hold some of them). They only show that certain particular arguments for the position are flawed, such as:

"Abortion is murder!" The archetypal murder is Charles Manson breaking into your house and shooting you. This sort of murder is bad for a number of reasons: you prefer not to die, you have various thoughts and hopes and dreams that would be snuffed out, your family and friends would be heartbroken, and the rest of society has to live in fear until Manson gets caught. If you define murder as "killing another human being", then abortion is technically murder. But it has none of the downsides of murder Charles Manson style. Although you can criticize abortion for many reasons, insofar as "abortion is murder" is an invitation to apply one's feelings in the Manson case directly to the abortion case, it ignores the latter's lack of the features that generated those intuitions in the first place2.

"Genetic engineering to cure diseases is eugenics!" Okay, you've got me there: since eugenics means "trying to improve the gene pool" that's clearly right. But what's wrong with eugenics? "What's wrong with eugenics? Hitler did eugenics! Those unethical scientists in the 1950s who sterilized black women without their consent did eugenics!" "And what was wrong with what Hitler and those unethical scientists did?" "What do you mean, what was wrong with them? Hitler killed millions of people! Those unethical scientists ruined people's lives." "And does using genetic engineering to cure diseases kill millions of people, or ruin anyone's life?" "Well...not really." "Then what's wrong with it?" "It's eugenics!"

"Evolutionary psychology is sexist!" If you define "sexist" as "believing in some kind of difference between the sexes", this is true of at least some evo psych. For example, Bateman's Principle states that in species where females invest more energy in producing offspring, mating behavior will involve males pursuing females; this posits a natural psychological difference between the sexes. "Right, so you admit it's sexist!" "And why exactly is sexism bad?" "Because sexism claims that men are better than women and that women should have fewer rights!" "Does Bateman's principle claim that men are better than women, or that women should have fewer rights?" "Well...not really." "Then what's wrong with it?" "It's sexist!"

A second, subtler use of the noncentral fallacy goes like this: "X is in a category whose archetypal member gives us an emotional reaction. Therefore, we should apply that same emotional reaction to X even if X gives some benefit that outweighs the harm."

"Capital punishment is murder!" Charles Manson-style murder is solely harmful. This kind of murder produces really strong negative feelings. The proponents of capital punishment believe that it might decrease crime, or have some other attending benefits. In other words, they believe it's "the good kind of murder"3, just like the introductory example concluded that Martin Luther King was "the good kind of criminal". But since normal murder is so taboo, it's really hard to take the phrase "the good kind of murder" seriously, and just mentioning the word "murder" can call up exactly the same amount of negative feelings we get from the textbook example.

"Affirmative action is racist!" True if you define racism as "favoring certain people based on their race", but once again, our immediate negative reaction to the archetypal example of racism (the Ku Klux Klan) cannot be generalized to an immediate negative reaction to affirmative action. Before we generalize it, we have to check first that the problems that make us hate the Ku Klux Klan (violence, humiliation, divisiveness, lack of a meritocratic society) are still there. Then, even if we do find that some of the problems persist (like disruption of meritocracy, for example) we have to prove that it doesn't produce benefits that outweigh these harms.

"Taxation is theft!" True if you define theft as "taking someone else's money regardless of their consent", but though the archetypal case of theft (breaking into someone's house and stealing their jewels) has nothing to recommend it, taxation (arguably) does. In the archetypal case, theft is both unjust and socially detrimental. Taxation keeps the first disadvantage, but arguably subverts the second disadvantage if you believe being able to fund a government has greater social value than leaving money in the hands of those who earned it. The question then hinges on the relative importance of these disadvantages. Therefore, you can't dismiss taxation without a second thought just because you have a natural disgust reaction to theft in general. You would also have to prove that the supposed benefits of this form of theft don't outweigh the costs.

Now, because most arguments are rapid-fire debate-club style, sometimes it's still useful to say "Taxation isn't theft!" At least it beats saying "Taxation is theft but nevertheless good", then having the other side say "Apparently my worthy opponent thinks that theft can be good; we here on this side would like to bravely take a stance against theft", and then having the moderator call time before you can explain yourself. If you're in a debate club, do what you have to do. But if you have the luxury of philosophical clarity, you would do better to forswear the Dark Arts and look a little deeper into what's going on.

Are there ever cases in which this argument pattern can be useful? Yes. For example, it may be a groping attempt to suggest a Schelling fence; for example, a principle that one must never commit theft even when it would be beneficial because that would make it harder to distinguish and oppose the really bad kinds of theft. Or it can be an attempt to spark conversation by pointing out a potential contradiction: for example "Have you noticed that taxation really does contain some of the features you dislike about more typical instances of theft? Maybe you never even thought about that before? Why do your moral intuitions differ in these two cases? Aren't you being kind of hypocritical?" But this usage seems pretty limited - once your interlocutor says "Yes, I considered that, but the two situations are different for reasons X, Y, and Z" the conversation needs to move on; there's not much point in continuing to insist "But it's theft!"

But in most cases, I think this is more of an emotional argument, or even an argument from "You would look silly saying that". You really can't say "Oh, he's the good kind of criminal", and so if you have a potentially judgmental audience and not much time to explain yourself, you're pretty trapped. You have been forced to round to the archetypal example of that word and subtract exactly the information that's most relevant.

But in all other cases, the proper response to being asked to subtract relevant information is "No, why should I?" - and that's why this is the worst argument in the world.

 

Footnotes

1: On advice from the community, I have deliberately included three mostly-liberal examples and three-mostly conservative examples, so save yourself the trouble of counting them up and trying to speculate on this article's biases.

2: This should be distinguished from deontology, the belief that there is some provable moral principle about how you can never murder. I don't think this is too important a point to make, because only a tiny fraction of the people who debate these issues have thought that far ahead, and also because my personal and admittedly controversial opinion is that much of deontology is just an attempt to formalize and justify this fallacy.

3: Some people "solve" this problem by saying that "murder" only refers to "non-lawful killing", which is exactly as creative a solution as redefining "criminal" to mean "person who breaks the law and is not Martin Luther King." Identifying the noncentral fallacy is a more complete solution: for example, it covers the related (mostly sarcastic) objection that "imprisonment is kidnapping".

4: EDIT 8/2013: I've edited this article a bit after getting some feedback and complaints. In particular I tried to remove some LW jargon which turned off some people who were being linked to this article but were unfamiliar with the rest of the site.

5: EDIT 8/2013: The other complaint I kept getting is that this is an uninteresting restatement of some other fallacy (no one can agree which, but poisoning the well comes up particularly often). The question doesn't seem too interesting to me - I never claimed particular originality, a lot of fallacies blend into each other, and the which-fallacy-is-which game isn't too exciting anyway - but for the record I don't think it is. Poisoning the well is a presentation of two different facts, such as "Martin Luther King was a plagiarist...oh, by the way, what do you think of Martin Luther King's civil rights policies?" It may have no relationship to categories, and it's usually something someone else does to you as a conscious rhetorical trick. Noncentral fallacy is presenting a single fact, but using category information to frame it in a misleading way - and it's often something people do to themselves. The above plagiarism example of poisoning the well is not noncentral fallacy. If you think this essay is about bog-standard poisoning the well, then either there is an alternative meaning to poisoning the well I'm not familiar with, or you are missing the point.

Comment author: [deleted] 22 August 2012 09:51:52PM 1 point [-]

I posted answers, so far as I have them, to your questions in the linked discussion.

Comment author: Yvain 23 August 2012 07:09:36AM 1 point [-]

Very many thanks!

Comment author: [deleted] 22 August 2012 12:42:37PM *  0 points [-]

Sorry, my summery left some important points in the dark. Aristotle is saying something like this:

When we say something is good, we say that it is a good X or Y, not good simpliciter. When we say someone is a good flute player, we don't mean that they are good and a flute player, but that they are good qua flute player. By picking someone out as a good flute player, we are already deploying some idea of what it is that distinctively they do and what it is that they aim to do qua flute player.

Now, we're presently asking ourselves what the good life is, and generally in ethics, we're concerned with what it means to be a good human being. So, when we say 'good human being', what idea are we deploying about what human beings distinctively do and aim at qua human beings? Well, it looks like the thing human beings distinctively do is reason, and the thing they do well when we say they are good qua human being is live a life of virtue in accordance with reason.

And the rest of my book is supposed to explain and elaborate on that.

Souldn't you skip straight to "If we think X is good, then X is good"?

This is kind of what he's saying. Aristotle isn't playing the "lets do ethics from scratch" game. He says explicitly that he is counting on his and his student's intuitions to get the whole thing going. Ethical philosophy, he says, is only possible for people who were raised well and generally have a pretty good idea about right and wrong and stuff. Aristotle's ethics is not about making people decent. You need that just to get started. Aristotle's ethics is about making people extraordinary. It's about greatness.

Comment author: Yvain 23 August 2012 05:56:30AM 0 points [-]

This is kind of what he's saying. Aristotle isn't playing the "lets do ethics from scratch" game. He says explicitly that he is counting on his and his student's intuitions to get the whole thing going. Ethical philosophy, he says, is only possible for people who were raised well and generally have a pretty good idea about right and wrong and stuff.

Really? I'm reading Ed Feser right now, and he's arguing very strongly that the only reason we don't have objectively grounded ethics right now is that we're not Aristotelians, and that the ability to objectively ground ethics is one of the biggest advantages of Aristotle over everyone else. He specifically attacks utilitarianism from an Aristotelian viewpoint for giving a moral criterion but not being able to prove from first principles that one should follow it.

Is he operating outside the Aristotelian mainstream?

Comment author: [deleted] 15 August 2012 02:50:39AM *  1 point [-]

He never addresses why a given purpose is the chief purpose, only that some intermediate goals are intended to further other goals.

I'm not sure I understand what you think the gap in his argument is (which is not to say it's not gappy). In I.7 he says specifically that human beings have a variety of aims and activities, but that a life in accordance with reason is the chief aim because it's the activity we pick out when we say someone is a good human being (as opposed to a good carpenter). So adresses, if inconclusively, the question 'Why is this our chief aim?' by pointing out that when we call a carpenter good, we mean that he's good at the activity of carpentry...in other words, the thing we picked out when we called him a 'carpenter'. In ethics, we're concerned with being a good human, so what activity do we pick out when we call someone simply a 'human'? Well, no one calls someone a good human being (in the ethical sense) for metabolizing or running well. We call them good for leading a life of reason and virtue. That's why a life of virtue is our chief aim: it's what we're good at when we're good at being human.

Also note that the whole run of I.7 has a hypothetical, provisional sort of character. He's clear that he thinks this whole 'chief aim of human life' thing is a serious and difficult problem. I.7 isn't a great argument, but he is at least taking the problem seriously.

To be honest, every time I try to read Aristotle, I end up drawing a triangular diagram and labeling the three points 'agape', 'hatred', and 'apathy', (in response to every time he uses the concept of "contraries"), and the fundamental theorem of calculus, if I'm reading the Physics.

I think a triangular diagram will be misleading in understanding Aristotle's theory of contrariety. A better one might be a continuous line, with a positive end and a negative end. Every value on the line is always expressed as a predicate of some subject characteristic of the contrariety. So the range of color, has as its characteristic substratum 'surface', and every color value on the line is predicated of a surface.

didn't peruse the entire NE to see if he addresses every possible answer, because I know well enough that "it will be evident that" his pet reason will be the One True Answer, and he will not address in enough detail why that is evident.

'It is evident that' is often a translation of a more colloquial and less committal segway. Aristotle's Greek diction, however horrible, is never as pompous as translations make it out to be.

Comment author: Yvain 22 August 2012 05:53:58AM 0 points [-]

I'm not sure I understand what you think the gap in his argument is (which is not to say it's not gappy). In I.7 he says specifically that human beings have a variety of aims and activities, but that a life in accordance with reason is the chief aim because it's the activity we pick out when we say someone is a good human being (as opposed to a good carpenter). So adresses, if inconclusively, the question 'Why is this our chief aim?' by pointing out that when we call a carpenter good, we mean that he's good at the activity of carpentry...in other words, the thing we picked out when we called him a 'carpenter'. In ethics, we're concerned with being a good human, so what activity do we pick out when we call someone simply a 'human'?

Is this argument circular? Or at least a disguised appeal to intuition?

"When we think of what a good human is, we think of a rational human. Therefore, what sets humans off is being rational. Therefore, the final cause of humans to be rational. Therefore, a good human is a rational human."

Couldn't you skip straight to "If we think X is good, then X is good"?

Comment author: palladias 14 August 2012 08:40:49PM 0 points [-]

A different way of summarizing telos that may be helpful when discussing this topic is that an object or agent fulfilling it's telos is supposed to be the most fully itself. So, for Aquinas and Aristotle, because the quality that most clearly sets off the category 'human' is "a reasoning animal" actions that interfere with that part of human identity are unnatural and interfere with telos.

This framing is perhaps a little easier to tie back to the last two rereads on Unnatural Categories and Magical Categories since then telos is linked to a partitioning of conceptspace.

One of the most interesting places I've seen "the x-iest x" outside of telos discussions is in Turing Tests, where one of the human controls wins a prize for "the most human human" (i.e. the human that was most frequently rated as human by the judges). There's a book out by one of the contestants, who set out to win that prize. Apparently, one fairly successful strategy is being belligerent. Not the most encouraging result, but interesting.

Comment author: Yvain 22 August 2012 05:51:04AM 2 points [-]

A different way of summarizing telos that may be helpful when discussing this topic is that an object or agent fulfilling it's telos is supposed to be the most fully itself. So, for Aquinas and Aristotle, because the quality that most clearly sets off the category 'human' is "a reasoning animal" actions that interfere with that part of human identity are unnatural and interfere with telos.

When Charlton Heston was on the Planet of the Apes and he found that human beings were no longer differentiated by their reasoning powers (which were sub-par) but by their hairlessness, should he have devoted his life to keeping exceptionally well-shaved?

(this question brought to you by my continuing confusion with teleology)

Comment author: Vladimir_M 14 August 2012 07:47:12AM *  13 points [-]

I think my problem with your responses on this thread so far has been that you've taken various liberal positions, said "Obviously this a sacredness value, liberals say it's about harm but they are lying", and not justified this.

"Lying" is not the right word, since it suggests conscious deception. The term I have used consistently is rationalization.

In order to demonstrate that liberal sexual values are sacredness rather than harm based, you'd need to point out some specific practice that was harmless but which liberals still violently opposed [...] or harmful but which liberals supported [...]

Arguing against liberal positions on such matters is very difficult because they tend to be backed by a vast arsenal of rationalizations based on purportedly rational considerations of harm or fairness, often coming from prestigious and accredited intellectual institutions where liberals predominate. This is of course in addition to the dense minefield of "boo lights" where an argument, whatever its real merits, will trigger such outrage in a liberal audience that the discourse will be destroyed and the speaker discredited.

So, while I can readily point out concrete examples of the sort you're asking, unfortunately in many of them, crossing the inferential distances would be an uphill battle, or there would be immediate unpleasantness that I'd rather avoid. Therefore I'll limit myself to a few more vague and general points:

  • Laissez-faire in sex leads to all kinds of expensive negative-sum signaling and other games. Why not crack down on those, which would lead to a clear improvement by any utilitarian metric?

  • If it's OK for the government to ban smoking and other activities harmful for public health, why not extend such treatment to sexual activities that have obvious and drastic public health implications?

  • If the alleged vast inequality of wealth is a legitimate complaint against economic laissez-faire, why is it not legitimate to complain about the vast inequality of sexual and romantic opportunities (and of the related social status) under sexual laissez-faire? (The problem is by no means limited to men, of course.)

  • Why the automatic hostility towards the idea that under sexual laissez-faire, a huge segment of the population, which lacks sufficient prudence and self-control, will make disastrous and self-destructive choices, so that restrictive traditional sexual norms may amount to a net harm reduction? Especially since liberals make analogous arguments in favor of paternalistic regulation of practically everything else.

There are many other examples too, but these are the best ones I can think of without either running into enormous inferential distances or sounding too provocative. It really seems to me that liberal norms change suddenly and dramatically towards laissez-faire once sexual matters come under consideration, and I don't see how this could be because their regular considerations of harm and fairness just happen to entail laissez-faire in this particular area and nowhere else.

I agree that certain liberal values are based on sacredness (diversity and anti-racism) or purity (environmentalism), although I have yet to hear any good argument that liberals explicitly value authority.

Explicitly, certainly not often. But in many of their observed views and behaviors, I detect strong authority-based intuitions, even though they will invariably be rationalized as something else. The typical way is to present authority as some kind of neutral and objective expertise, even in areas where this makes no sense.

Once Peter Singer says he can't really see any problems with infanticide because it doesn't harm anyone, the hypothesis that he still is secretly trying to uphold sacredness values just as much as everyone else becomes pretty hard to support.

As I said, I'm not an expert on Singer in particular, and I don't deny the possibility that he might be an outlier in this regard. (Although I do remember reading things from him that seemed to me like a clear case of rationalizing fundamentally non-utilitarian liberal positions.) Also, I agree that someone's serious utilitarian bullet-biting on some issues provides some evidence that he is overall less dedicated to the values of sacredness etc. I do think, however, that you underestimate how often such serious bullet-biters can be inconsistent on other issues.

Comment author: Yvain 15 August 2012 05:19:38AM *  11 points [-]

You're right, I shouldn't have used the word "lying". That mistake bothers me when other people do it, and I'm sorry for doing it myself.

But other than that...I'm afraid the whole point of my last post was to ask for examples, that we have different standards of what constitutes an example, and that I'm still not happy. For me, "Liberals have strong norms around equality" is not an example; I'm thinking something more along the lines of "You know how liberals are pro-choice? That's irrational for reasons X and Y and Z."

Laissez-faire in sex leads to all kinds of expensive negative-sum signaling and other games. Why not crack down on those, which would lead to a clear improvement by any utilitarian metric?

Can you give an example of a specific laissez-faire sexual policy that causes expensive negative-sum signaling games, and a practically workable less laissez-faire policy that would solve those negative-sum signaling games?

If it's OK for the government to ban smoking and other activities harmful for public health, why not extend such treatment to sexual activities that have obvious and drastic public health implications?

Can you give an example of a sexual activity that has such obvious and drastic public health implications that it should be banned?

If the alleged vast inequality of wealth is a legitimate complaint against economic laissez-faire, why is it not legitimate to complain about the vast inequality of sexual and romantic opportunities (and of the related social status) under sexual laissez-faire?

It doesn't seem illegitimate to complain about it. What particular policies are you recommending?

Why the automatic hostility towards the idea that under sexual laissez-faire, a huge segment of the population, which lacks sufficient prudence and self-control, will make disastrous and self-destructive choices, so that restrictive traditional sexual norms may amount to a net harm reduction?

You're assuming the conclusion when you say "automatic hostility". If you gave examples of a traditional norm that solved this problem, I would have be able to form more of an opinion on whether that traditional norm was genuinely harm-reducing.

Explicitly, certainly not often. But in many of their observed views and behaviors, I detect strong authority-based intuitions, even though they will invariably be rationalized as something else. The typical way is to present authority as some kind of neutral and objective expertise, even in areas where this makes no sense.

Can you give an example of a liberal intuition which is authority-based but gets rationalized away to something else?

I do think, however, that you underestimate how often such serious bullet-biters can be inconsistent on other issues.

Can you give an example of a serious bullet-biter being inconsistent on other issues?

I hate to sound like a broken record here, it's just that anyone supporting any position at all can say "All my opponents really hold their positions for terrible reasons, and all their seemingly-good arguments are really just rationalizations". In the absence of specific evidence, this is just an assertion, and not an uncommon one.

Even though I have some pretty good guesses what you mean by some of these, I don't want to find myself straw-manning you by accident just because it's easy for me to come up with examples I can refute.

I understand if you don't want to start a brouhaha by posting controversial positions publicly. If you want to private message me an example or two, I'm usually pretty hard to offend, and I promise not to share it without your permission.

Comment author: Cosmos 14 August 2012 09:58:24PM *  6 points [-]

Yvain, it seems like some of this is potentially answered by how this interacts with other cognitive biases present.

Re: specific delusions, when you have an entire class of equally-explanatory hypotheses, how do you choose between them? The availability heuristic! These hypotheses do have to come from somewhere inside the neural network after all. You could argue that availability is a form of "priors", but these "priors" are formed on the level of neurons themselves and not a specific brain region: some connection strengths are stronger than others.

I would not wish brain damage on anyone, but should one of us have that unfortunate circumstance befall us I would be extremely inclined to go talk to them. I am so damn curious what this feels like from the inside! I am somewhat embarrassed to admit that the thought of having to build completely new neural connections to get around existing damage sounds like an insanely interesting challenge...

I also wonder about reasoning our way out of delusional states. The closest parallel that most people have access to would be the use of various psychoactives. I have heard multiple reports of people who have reasoned their way out of delusional conclusions on cannabinoid agonists and 5-HT2A agonists (and dopamine agonists, with lesser evidence).

The most difficult challenge would appear to be kappa opioid agonism, a dissociative state induced by the federally-legal herb salvia divinorum. Most users report being unaware they ingested a substance at all, no awareness of having a body, and no concept of self-identity, coincident with extreme perceptual distortions. I am no longer clear what Bayesian reasoning would even look like for some points in mindspace.

Edit: I thought of another relevant state: delirium induced by anticholinergics. Unlike 5-HT2A agonists where people do not confuse perceptual distortions for reality, in delirious states people do routinely believe that what they are perceiving is actually occurring. Unfortunately these states are widely regarded as unpleasant, and no rationalist I know personally has experimented with sufficiently large doses of anticholinergics.

Comment author: Yvain 15 August 2012 02:39:59AM 3 points [-]

Availability heuristic seems related, but still doesn't explain why delusions are so much more fixed than ordinary conclusions.

I think dreams are also a good parallel for psychosis, but it's hard to tell how good without having been psychotic.

Comment author: Cosmos 14 August 2012 09:41:51PM 6 points [-]

I had the exact same thought myself back in 2008, so I asked an experimental psych professor about this. At the same, he said that the TMS devices that we had are somewhat wide-area and also induce considerable muscle activation. This doesn't matter very much when studying the occipital lobe, but for the prefrontal cortex you basically start scrunching up the person's face, which is fairly distracting. Maybe worth trying anyway.

I've wanted to get my hands on a TMS device for years. Building one at home does not seem particularly feasible, and the magnetism involved is probably dangerous for nearby metal/electronics...

Comment author: Yvain 15 August 2012 02:35:49AM 1 point [-]

I don't know the technical differences between TMS and TDCS, but http://flowstateengaged.com/ looks promising.

Comment author: Epiphany 13 August 2012 04:07:54AM *  9 points [-]

Individual intelligence differences are NOT thought of as okay. Try introducing yourself on a random message board with each of these and see what happens:

  • Hi, I'm Joe and the main thing I'm good at is art.
  • Hi, I'm Joe and I'm proud of my Native American ancestry.
  • Hi, I'm Joe and my IQ is 170.

Joe with the IQ of 170 will be called arrogant, a liar, an elitist, treated like a scam artist, or told he has no social skills. That's not telling Joe he's okay. That's telling Joe not to talk about his difference. Let's explore what it means to be told you can't talk about your difference for a moment. Imagine going into a room and saying each of the following:

  • Hey, don't say you've got Native American blood, that's socially inept.

^ This comment will surely be interpreted as racism.

  • Hey, don't say you're good at art, you're a liar.

^ This comment will be interpreted as an extremely rude or even oppressive comment. Making judgments about whether artists are "good" or "bad" is taboo and considered, by many, to be oppressive to self-expression.

  • Hey, don't say your IQ is 170, don't be an elitist.

^ This comment prejudges the person. It assumes that they're an elitist when they're just talking about an intellectual difference that doesn't prove anything about your personality.

So, why doesn't Joe get to have the same freedom to express himself without society oppressing that? Why doesn't he get to talk about his difference without expecting prejudiced remarks that jump to conclusions about who he is?

We have a million excuses for this. "People feel threatened by intellect." Well, they used to feel threatened by black people, but that doesn't excuse society from working on removing their prejudices about black people and it doesn't excuse them from working on removing their prejudices about gifted people.

"That's just not polite." <- This is an interesting excuse. I'll explain why:

Imagine you go into a room and say "Hi, I'm white." (I realize that people of any race may read this comment, I am asking you to humor my hypothetical situation for a moment.)

Your race is evident. This is a neutral statement of fact.

If someone tells you "That's just not polite." why are they saying that? They're probably confusing it with an expression of the white pride attitude that is associated with the KKK. They're assuming that you're prejudiced.

What if you went up to a bunch of random white people and accused them of hating black people? Since this doesn't happen frequently, they'd probably be mostly bewildered. But imagine if random people did that to them every day.

Prejudice is a very serious offense to be accused of. It would stress them out. They'd wonder what kinds of social and career opportunities they might be missing out on. They might become more cautious to guard their physical safety - after all, prejudice is the kind of thing people get really heated about and some people get violent when they're upset. They'd start to hide hints that they're white on things like resumes. They would be oppressed by an assumption that they're prejudiced, just the same way that they'd be oppressed by an assumption that they're all criminals.

Accusing a person of prejudice simply for being part of a certain group is, in and of itself, prejudiced. That's prejudging them based on some trait that they can't control, not on their behavior. Yet, if you claim to have a high IQ, you are very likely to be accused of elitism. People act like this prejudice against people with a high IQ is okay and that gifted people should behave like an oppressed minority by hiding their difference.

I'm glad you think it's okay with the rest of the world for people to talk about their intelligence differences, I think that's okay. But a looooooot of people don't!

Comment author: Yvain 14 August 2012 06:03:44AM 7 points [-]

The most important difference here is that the first two statements, in addition to being boasts, also convey a non-boasting fact about the particular area you are interested in. For example "I'm good at art" strikes me less as conveying information about being especially talented, as saying that art is the particular subject you like and work on.

Compare someone who goes into an artists' workshop or an art class or something. They introduce themselves with "Hi, I'm Joe, and I'm really good at art." Now it is boasting. Everyone there is interested in art, and Joe is making a claim of being especially good at it compared to all the other artists. (This is even more true if we add some kind of number or statistic to it. "Hi, I'm Joe, and I'm in the 99th percentile for art skills.". Now he's definitely boasting, since the statistic doesn't do anything to help describe his interest.)

Intelligence is very general, and it's something you have rather than something you're interested in. That might make claims to it seem more boastful.

Comment author: Vladimir_M 14 August 2012 03:54:38AM *  8 points [-]

Fallacy of gray? Arguably no one has completely removed all minor unconscious belief in purity/sanctity/authority based values, but I think endorsing harm/fairness values at least correlates with holding fewer values based on P/S/A, even secretly.

There are two distinct questions here:

  1. Are the standard liberal ideological positions (in the American sense of the word) really as low on the sacredness/authority/in-group values as Haidt would claim?

  2. Are there, generally speaking, significant numbers of people (perhaps weighted by their influence) whose ideological positions are truly low on the sacredness/authority/in-group values? (Whatever their overlap with the standard liberal positions might be.)

I believe that the answer to (1) is decisively no. And here I don't have in mind some minor holdovers, but some of the very central tenets of the ideology of modern liberalism -- which are largely liberal innovations, and not just unexamined baggage from the past. So even if I'm committing fallacies here, they're not fallacies of gray. In this thread and the linked older comments, I have already elaborated on one significant example where the standard liberal positions are heavy on sacredness (the sacralization of individual autonomy in sex-related matters). I could also give examples of liberal authority and in-group values, some of which I've already mentioned in passing. Unfortunately, you can probably see why such topics are, practically by definition, inordinately likely to inflame passions and destroy the discourse.

As for (2), clearly, if you look for outliers hard enough, you'll find them, and there is some variability even among people closer to the mainstream. But I think that you are greatly underestimating how much of the entire utilitarianism shtick in the contemporary ideological debates is just a convenient framework for rationalizations of views and intuitions held for completely different reasons. (And it's not very different for egalitarian and other arguments that leverage the fairness intuitions.)

Even when it comes to bullet-biters who will be convinced by utilitarian (or egalitarian etc.) arguments to adopt odd and extreme positions on some issues, it's a mistake to conclude from this that they have done an equally consistent scrutiny of all their beliefs, or even the majority of them. I think this is a good description of someone like Singer (with the caveat that I haven't read anything close to a large and representative sample of his work, so that my view of his particular case might be biased).

Comment author: Yvain 14 August 2012 05:19:07AM *  13 points [-]

I think my problem with your responses on this thread so far has been that you've taken various liberal positions, said "Obviously this a sacredness value, liberals say it's about harm but they are lying", and not justified this. Or else "Some people say they are utilitarians, but obviously they are lying and have sacredness and purity and authority values just like everyone else" and not justified that either.

For example, where exactly is this liberal sacredness around sexual autonomy? The place I see liberals really get worked up about this is tolerance of homosexuality, but the standard liberal mantra in this case, that it's okay because it "doesn't harm anyone", seems to me to be entirely correct - it's throwing out a conservative purity-based value in favor of a genuinely harm-based value. Liberals are pretty happy to oppose clear-cut cases of harm in sexual relations like rape or lying about STDs, not to mention that most of them oppose pedophilia and prostitution.

In order to demonstrate that liberal sexual values are sacredness rather than harm based, you'd need to point out some specific sexual practice that was harmless but which liberals still violently opposed (arranged marriage? Do liberals have a strong opinion on this?) or harmful but which liberals supported (maybe no-fault divorce? But this is far from universally-supported among liberals, it's far from clear that it's harmful, and I don't think most liberals who do support it refer to a principle of sexual autonomy or have the fervor that tends to characterize sacred values.)

Overall I think liberal support for sexual autonomy, insofar as it's a useful idea at all, to be mostly based around autonomy values (obviously), harm values (as the liberals themselves say), and maybe an overreaction to really disliking conservative values around things like homosexuality or sexual "prudery". I think you have further to go in demonstrating that there's really a strong foundation of sacredness there, although I understand if you don't want to turn this thread into a debate on sex mores.

I agree that certain liberal values are based on sacredness (diversity and anti-racism) or purity (environmentalism), although I have yet to hear any good argument that liberals explicitly value authority. But two examples, both of which are polluted with confounders (racism really is really harmful), hardly seem like enough to say they are just as interested in these values as conservatives and totally deceiving themselves when they say they aren't.

And I have the same objections to your comments on libertarians and utilitarians. Yeah, only a few percent of the population is either (although it's more in places where people are genuinely interested in philosophical and political issues and likely to think for themselves, and only about 20% of Americans self-identify as "liberal" anyway). But libertarians for example seem ruthlessly consistent in opposing government intervention into any area (except maybe defense and policing), and I have a higher opinion of utilitarians than you do. Once Peter Singer says he can't really see any problems with infanticide because it doesn't harm anyone, the hypothesis that he still is secretly trying to uphold sacredness values just as much as everyone else becomes pretty hard to support.

Similarly, not every case of hypocrisy is a case of secretly having sacredness or purity values. I don't fail at efficient charity because I secretly believe that inefficient charity is sacred. I fail at efficient charity because utilitarianism is really hard.

Comment author: Vladimir_M 12 August 2012 06:53:44AM *  22 points [-]

It is, however, missing a piece: why are there people who don't share all five foundations?

You are right that Haidt is missing that piece, although judging by his recent writings, he might be slowly converging towards the answer. Namely, the answer is that, contrary to Haidt's model of contemporary ideologies, there are in fact no such people.

What does exist are people whose ideology says that harm and (maybe) fairness are the only rational and reasonable moral foundations, while the other ones are only due to ignorance, stupidity, backwardness, malice, etc. Nevertheless, these same people have their own strong norms of sacredness, purity, authority, and in-group loyalty, for which they however invent ideologically motivated rationalizations in terms of harm and fairness. These rationalizations are usually very flimsy, and often they amount to little more than an instinctive emotional urge to dismiss anyone who asks unpleasant questions as crazy or malicious. Yet, given the high status and institutional dominance of such ideologies, their adherents generally do manage to create a public image of themselves as concerned only with the "rational" foundations (and thus superior intellectually and morally to their ideological opponents).

As for the claim that "you need loyalty, authority and sanctity to run a decent society," I would actually go further and say that they are necessary for any sort of organized human society. In fact, the claim can be stated even more strongly: since humans are social beings who can live and reproduce only within organized societies, these things simply will exist wherever there are humans. Therefore, if you are concerned with harm, the only reasonable question you can ask is about the practical consequences of the (necessarily multi-foundational) social norms in different societies on whatever metric you use to evaluate harm. And here you will find that, even in terms of a purely utilitarian metric, an accurate analysis of the social role of the norms based on these "irrational" foundations will give you very different answers from those given by the pseudo-rational ideologies that claim to reject these foundations.

Comment author: Yvain 14 August 2012 02:08:44AM *  6 points [-]

Fallacy of gray? Arguably no one has completely removed all minor unconscious belief in purity/sanctity/authority based values, but I think endorsing harm/fairness values at least correlates with holding fewer values based on P/S/A, even secretly.

I am also not clear whether you're saying only that mainstream large liberal parties like UK Labor or US Democrats secretly have many P/S/A values, or whether you would say the same is true of people like Peter Singer or the more pragmatic/less ideological strains of libertarian. I think the gradient from the Pope to Nancy Pelosi to Peter Singer is quite clear, even if the last might still have some P/S/A values lurking somewhere.

If you disagree, can you name a few purity, sanctity, or authority based values you expect intelligent liberals or libertarians on LW to endorse?

Comment author: Spurlock 13 August 2012 12:37:53PM 3 points [-]

but it's also impossible to convince him he's Alexander the Great (at least I think so; I don't know if it's ever been tried).

At the very least (pretending that there are no ethical concerns), it seems that you ought to be able to exaggerate a patient's delusions. "We ran some tests, and it turns out that you're Jesus, John Lennon, and George Washington!".

To this same question, I can't help but notice that the brain damage being discussed is right-side brain aka "revolutionary" brain damage. So if it turns out that it isn't possible to get a paranoid patient to switch from FBI to KGB, it might simply be a case of inability to discard hypotheses (it seems like the original delusion, the CIA, wouldn't count because for most of us "the CIA isn't following me" isn't an explicit belief). But then, I am not a neurologist or psychologist, so the pool of data I'm working with is 100% limited to that which has been written about by Yvain on LW :-)

Comment author: Yvain 14 August 2012 01:44:33AM *  10 points [-]

The patient who believes he is Jesus and John Lennon will pretty much agree he is any famous figure you mention to him, but he never seems to make a big deal of it, whereas those two are the ones he's always going on about.

Comment author: Vaniver 13 August 2012 06:23:33PM *  6 points [-]

It seems improbable, but I recently heard about an n=1 personal experiment of a rationalist with schizophrenia who used successfully used Bayes to convince themselves that a delusion (or possibly hallucination; the story was unclear) was false. I don't have their permission to post their story here, but I hope they'll appear in the comments.

I was under the impression that learning to recognize hallucinations was a standard component of schizophrenia therapy.

Comment author: Yvain 14 August 2012 01:41:34AM 0 points [-]

Therapists can very very carefully try to talk patients out of their delusions, but I've always heard of it as a complicated long-term process and I've never before heard of Bayes being used directly.

Comment author: Eugine_Nier 14 August 2012 12:03:52AM *  2 points [-]

There must be some fundamental difference between how one draws inferences from mental states versus everything else.

Talking about "drawing inferences from mental states" strikes me as a case of the homunculus fallacy, i.e., thinking that there's some kind of homunculus sitting inside our brains looking at the mental states and drawing inferences. Whereas in reality mental states are inferences.

Comment author: Yvain 14 August 2012 01:40:29AM 2 points [-]

Really? I don't see that at all. The same mental state can be both an inference and a premise for the next inference. For example, "I feel really tired lately -> Maybe I'm sick" seems pretty straightforward, as does "I am a guy and feel really attracted to other guys -> maybe I'm gay".

Comment author: torekp 14 August 2012 12:40:56AM 3 points [-]

Do you have any evidence of brain damage in schizophrenia that isn't explainable by drug use (including antipsychotics especially) and is fairly common among schizophrenics?

Regarding arguing oneself out of delusion, cognitive therapy for schizophrenia has a decent track record. More info on request, after my wife gets home (she's a psychologist).

Comment author: Yvain 14 August 2012 01:38:34AM 2 points [-]

See for example http://www.schizophrenia.com/research/schiz.brain.htm on structural brain damage. For functional brain damage, read the above-linked paper by McKay where he starts talking about change in patterns of prediction error signal activation in the right prefrontal cortex.

Bayes for Schizophrenics: Reasoning in Delusional Disorders

88 Post author: Yvain 13 August 2012 07:22PM

Related to: The Apologist and the Revolutionary, Dreams with Damaged Priors

Several years ago, I posted about V.S. Ramachandran's 1996 theory explaining anosognosia through an "apologist" and a "revolutionary".

Anosognosia, a condition in which extremely sick patients mysteriously deny their sickness, occurs during right-sided brain injury but not left-sided brain injury. It can be extraordinarily strange: for example, in one case, a woman whose left arm was paralyzed insisted she could move her left arm just fine, and when her doctor pointed out her immobile arm, she claimed that was her daughter's arm even though it was obviously attached to her own shoulder. Anosognosia can be temporarily alleviated by squirting cold water into the patient's left ear canal, after which the patient suddenly realizes her condition but later loses awareness again and reverts back to the bizarre excuses and confabulations.

Ramachandran suggested that the left brain is an "apologist", trying to justify existing theories, and the right brain is a "revolutionary" which changes existing theories when conditions warrant. If the right brain is damaged, patients are unable to change their beliefs; so when a patient's arm works fine until a right-brain stroke, the patient cannot discard the hypothesis that their arm is functional, and can only use the left brain to try to fit the facts to their belief.

In the almost twenty years since Ramachandran's theory was published, new research has kept some of the general outline while changing many of the specifics in the hopes of explaining a wider range of delusions in neurological and psychiatric patients. The newer model acknowledges the left-brain/right-brain divide, but adds some new twists based on the Mind Projection Fallacy and the brain as a Bayesian reasoner.


INTRODUCTION TO DELUSIONS

Strange as anosognosia is, it's only one of several types of delusions, which are broadly categorized into polythematic and monothematic. Patients with polythematic delusions have multiple unconnected odd ideas: for example, the famous schizophrenic game theorist John Nash believed that he was defending the Earth from alien attack, that he was the Emperor of Antarctica, and that he was the left foot of God. A patient with a monothematic delusion, on the other hand, usually only has one odd idea. Monothematic delusions vary less than polythematic ones: there are a few that are relatively common across multiple patients. For example:

In the Capgras delusion, the patient, usually a victim of brain injury but sometimes a schizophrenic, believes that one or more people close to her has been replaced by an identical imposter. For example, one male patient expressed the worry that his wife was actually someone else, who had somehow contrived to exactly copy his wife's appearance and mannerisms. This delusion sounds harmlessly hilarious, but it can get very ugly: in at least one case, a patient got so upset with the deceit that he murdered the hypothesized imposter - actually his wife.

The Fregoli delusion is the opposite: here the patient thinks that random strangers she meets are actually her friends and family members in disguise. Sometimes everyone may be the same person, who must be as masterful at quickly changing costumes as the famous Italian actor Fregoli (inspiring the condition's name).

In the Cotard delusion, the patient believes she is dead. Cotard patients will neglect personal hygiene, social relationships, and planning for the future - as the dead have no need to worry about such things. Occasionally they will be able to describe in detail the "decomposition" they believe they are undergoing.

Patients with all these types of delusions1 - as well as anosognosiacs - share a common feature: they usually have damage to the right frontal lobe of the brain (including in schizophrenia, where the brain damage is of unknown origin and usually generalized, but where it is still possible to analyze which areas are the most abnormal). It would be nice if a theory of anosognosia also offered us a place to start explaining these other conditions, but this Ramachandran's idea fails to do. He posits a problem with belief shift: going from the originally correct but now obsolete "my arm is healthy" to the updated "my arm is paralyzed". But these other delusions cannot be explained by simple failure to update: delusions like "the person who appears to be my wife is an identical imposter" never made sense. We will have to look harder.

ABNORMAL PERCEPTION: THE FIRST FACTOR

Coltheart, Langdon, and McKay posit what they call the "two-factor theory" of delusion. In the two-factor theory, one problem causes an abnormal perception, and a second problem causes the brain to come up with a bizarre instead of a reasonable explanation.

Abnormal perception has been best studied in the Capgras delusion. A series of experiments, including some by Ramachandran himself, demonstrate that Capgras patients lack a skin conductance response (usually used as a proxy of emotional reaction) to familiar faces. This meshes nicely with the brain damage pattern in Capgras, which seems to involve the connection between the face recognition areas in the temporal lobe and the emotional areas in the limibic system. So although the patient can recognize faces, and can feel emotions, the patient cannot feel emotions related to recognizing faces.

The older "one-factor" theories of delusion stopped here. The patient, they said, knows that his wife looks like his wife, but he doesn't feel any emotional reaction to her. If it was really his wife, he would feel something - love, irritation, whatever - but he feels only the same blankness that would accompany seeing a stranger. Therefore (the one-factor theory says) his brain gropes for an explanation and decides that she really is a stranger. Why does this stranger look like his wife? Well, she must be wearing a very good disguise.

One-factor theories also do a pretty good job of explaining many of the remaining monothematic delusions. A 1998 experiment shows that Cotard delusion sufferers have a globally decreased autonomic response: that is, nothing really makes them feel much of anything - a state consistent with being dead. And anosognosiacs have lost not only the nerve connections that would allow them to move their limbs, but the nerve connections that would send distress signals and even the connections that would send back "error messages" if the limb failed to move correctly - so the brain gets data that everything is fine.

The basic principle behind the first factor is "Assume that reality is such that my mental states are justified", a sort of Super Mind Projection Fallacy.

Although I have yet to find an official paper that says so, I think this same principle also explains many of the more typical schizophrenic delusions, of which two of the most common are delusions of grandeur and delusions of persecution. Delusions of grandeur are the belief that one is extremely important. In pop culture, they are typified by the psychiatric patient who believes he is Jesus or Napoleon - I've never met any Napoleons, but I know several Jesuses and recently worked with a man who thought he was Jesus and John Lennon at the same time. Here the first factor is probably an elevated mood (working through a miscalibrated sociometer). "Wow, I feel like I'm really awesome. In what case would I be justified in thinking so highly of myself? Only if I were Jesus and John Lennon at the same time!" A similar mechanism explains delusions of persecution, the classic "the CIA is after me" form of disease. We apply the Super Mind Projection Fallacy to a garden-variety anxiety disorder: "In what case would I be justified in feeling this anxious? Only if people were constantly watching me and plotting to kill me. Who could do that? The CIA."

But despite the explanatory power of the Super Mind Projection Fallacy, the one-factor model isn't enough.

ABNORMAL BELIEF EVALUATION: THE SECOND FACTOR

The one-factor model requires people to be really stupid. Many Capgras patients were normal intelligent people before their injuries. Surely they wouldn't leap straight from "I don't feel affection when I see my wife's face" to "And therefore this is a stranger who has managed to look exactly like my wife, sounds exactly like my wife, owns my wife's clothes and wedding ring and so on, and knows enough of my wife's secrets to answer any question I put to her exactly like my wife would." The lack of affection vaguely supports the stranger hypothesis, but the prior for the stranger hypothesis is so low that it should never even enter consideration (remember this phrasing: it will become important later.) Likewise, we've all felt really awesome at one point or another, but it's never occurred to most of us that maybe we are simultaneously Jesus and John Lennon.

Further, most psychiatric patients with the deficits involved don't develop delusions. People with damage to the ventromedial area suffer the same disconnection between face recognition and emotional processing as Capgras patients, but they don't draw any unreasonable conclusions from it. Most people who get paralyzed don't come down with anosognosia, and most people with mania or anxiety don't think they're Jesus or persecuted by the CIA. What's the difference between these people and the delusional patients?

The difference is the right dorsolateral prefrontal cortex, an area of the brain strongly associated with delusions. If whatever brain damage broke your emotional reactions to faces or paralyzed you or whatever spared the RDPC, you are unlikely to develop delusions. If your brain damage also damaged this area, you are correspondingly more likely to come up with a weird explanation.

In his first papers on the subject, Coltheart vaguely refers to the RDPC as a "belief evaluation" center. Later, he gets more specific and talks about its role in Bayesian updating. In his chronology, a person damages the connection between face recognition and emotion, and "rationally" concludes the Capgras hypothesis. In his model, even if there's only a 1% prior of your spouse being an imposter, if there's a 1000 times greater likelihood of you not feeling anything toward an imposter than to your real spouse, you can "rationally" come to believe in the delusion. In normal people, this rational belief then gets worn away by updating based on evidence: the imposter seems to know your spouse's personal details, her secrets, her email passwords. In most patients, this is sufficient to have them update back to the idea that it is really their spouse. In Capgras patients, the damage to the RDPC prevents updating on "exogenous evidence" (for some reason, the endogenous evidence of the lack of emotion itself still gets through) and so they maintain their delusion.

This theory has some trouble explaining why patients are still able to update about other situations, but Coltheart speculates that maybe the belief evaluation system is weakened but not totally broken, and can deal with anything except the ceaseless stream of contradictory endogenous information.

EXPLANATORY ADEQUACY BIAS

McKay makes an excellent critique of several questionable assumptions of this theory.

First, is the Capgras hypothesis ever plausible? Coltheart et al pretend that the prior is 1/100, but this implies that there is a base rate of your spouse being an imposter one out of every hundred times you see her (or perhaps one out of every hundred people has a fake spouse) either of which is preposterous. No reasonable person could entertain the Capgras hypothesis even for a second, let alone for long enough that it becomes their working hypothesis and develops immunity to further updating from the broken RDPC.

Second, there's no evidence that the ventromedial patients - the ones who lose face-related emotions but don't develop the Capgras delusion - once had the Capgras delusion but then successfully updated their way out of it. They just never develop the delusion to begin with.

McKay keeps the Bayesian model, but for him the second factor is not a deficit in updating in general, but a deficit in the use of priors. He lists two important criteria for reasonable belief: "explanatory adequacy" (what standard Bayesians call the likelihood ratio; the new data must be more likely if the new belief is true than if it is false) and "doxastic conservativism" (what standard Bayesians call the prior; the new belief must be reasonably likely to begin with given everything else the patient knows about the world).

Delusional patients with damage to their RDPC lose their ability to work with priors and so abandon all doxastic conservativism, essentially falling into a what we might term the Super Base Rate Fallacy. For them the only important criterion for a belief is explanatory adequacy. So when they notice their spouse's face no longer elicits any emotion, they decide that their spouse is not really their spouse at all. This does a great job of explaining the observed data - maybe the best job it's possible for an explanation to do. Its only minor problem is that it has a stupendously low prior, and this doesn't matter because they are no longer able to take priors into account.

This also explains why the delusional belief is impervious to new evidence. Suppose the patient's spouse tells personal details of their honeymoon that no one else could possibly know. There are several possible explanations: the patient's spouse really is the patient's spouse, or (says the left-brain Apologist) the patient's spouse is an alien who was able to telepathically extract the relevant details from the patient's mind. The telepathic alien imposter hypothesis has great explanatory adequacy: it explains why the person looks like the spouse (the alien is a very good imposter), why the spouse produces no emotional response (it's not the spouse at all) and why the spouse knows the details of the honeymoon (the alien is telepathic). The "it's really your spouse" explanation only explains the first and the third observations. Of course, we as sane people know that the telepathic alien hypothesis has a very low base rate plausibility because of its high complexity and violation of Occam's Razor, but these are exactly the factors that the RDPC-damaged2 patient can't take into account. Therefore, the seemingly convincing new evidence of the spouse's apparent memories only suffices to help the delusional patient infer that the imposter is telepathic.

The Super Base Rate Fallacy can explain the other delusional states as well. I recently met a patient who was, indeed, convinced the CIA were after her; of note she also had extreme anxiety to the point where her arms were constantly shaking and she was hiding under the covers of her bed. CIA pursuit is probably the best possible reason to be anxious; the only reason we don't use it more often is how few people are really pursued by the CIA (well, as far as we know). My mentor warned me not to try to argue with the patient or convince her that the CIA wasn't really after her, as (she said from long experience) it would just make her think I was in on the conspiracy. This makes sense. "The CIA is after you and your doctor is in on it" explains both anxiety and the doctor's denial of the CIA very well; "The CIA is not after you" explains only the doctor's denial of the CIA. For anyone with a pathological inability to handle Occam's Razor, the best solution to a challenge to your hypothesis is always to make your hypothesis more elaborate.

OPEN QUESTIONS


Although I think McKay's model is a serious improvement over its predecessors, there are a few loose ends that continue to bother me.

"You have brain damage" is also a theory with perfect explanatory adequacy. If one were to explain the Capgras delusion to Capgras patients, it would provide just as good an explanation for their odd reactions as the imposter hypothesis. Although the patient might not be able to appreciate its decreased complexity, they should at least remain indifferent between the two hypotheses. I've never read of any formal study of this, but given that someone must have tried explaining the Capgras delusion to Capgras patients I'm going to assume it doesn't work. Why not?

Likewise, how come delusions are so specific? It's impossible to convince someone who thinks he is Napoleon that he's really just a random non-famous mental patient, but it's also impossible to convince him he's Alexander the Great (at least I think so; I don't know if it's ever been tried). But him being Alexander the Great is also consistent with his observed data and his deranged inference abilities. Why decide it's the CIA who's after you, and not the KGB or Bavarian Illuminati?

Why is the failure so often limited to failed inference from mental states? That is, if a Capgras patient sees it is raining outside, the same process of base rate avoidance that made her fall for the Capgras delusion ought to make her think she's been transported to ther rainforest or something. This happens in polythematic delusion patients, where anything at all can generate a new delusion, but not those with monothematic delusions like Capgras. There must be some fundamental difference between how one draws inferences from mental states versus everything else.

This work also raises the question of whether one can one consciously use System II Bayesian reasoning to argue oneself out of a delusion. It seems improbable, but I recently heard about an n=1 personal experiment of a rationalist with schizophrenia who used successfully used Bayes to convince themselves that a delusion (or possibly hallucination; the story was unclear) was false. I don't have their permission to post their story here, but I hope they'll appear in the comments.

FOOTNOTES


1: I left out discussion of the Alien Hand Syndrome, even though it was in my sources, because I believe it's more complicated than a simple delusion. There's some evidence that the alien hand actually does move independently; for example it will sometimes attempt to thwart tasks that the patient performs voluntarily with their good hand. Some sort of "split brain" issues seem like a better explanation than simple Mind Projection.

2: The right dorsolateral prefrontal cortex also shows up in dream research, where it tends to be one of the parts of the brain shut down during dreaming. This provides a reasonable explanation of why we don't notice our dreams' implausibility while we're dreaming them - and Eliezer specifically mentions he can't use priors correctly in his dreams. It also highlights some interesting parallels between dreams and the monothematic delusions. For example, the typical "And then I saw my mother, but she was also somehow my fourth grade teacher at the same time" effect seems sort of like Capgras and Fregoli. Even more interestingly, the RDPC gets switched on during lucid dreaming, providing an explanation of why lucid dreamers are able to reason normally in dreams. Because lucid dreaming also involves a sudden "switching on" of "awareness", this makes the RDPC a good target area for consciousness research.

Comment author: Yvain 10 August 2012 02:11:43AM *  8 points [-]

By request, I've put up a backup copy of my entry at http://www.raikoth.net/Stuff/LessWrong/mineral.doc .

Comment author: RobertLumley 01 August 2012 06:32:39PM *  1 point [-]

It's getting close to a year since we did the last census of LW, (Results) (I actually thought it had been longer until I checked) Is it time for another one? I think about once a year is right, but we may be growing or changing fast enough that more than that is appropriate. Ergo, a poll:

Edit: If you're rereading the results and have suggestions for how to improve the census, it might be a good idea to reply to this comment.

Comment author: Yvain 01 August 2012 08:05:22PM 11 points [-]

I was planning to do one in October of this year (though now that it's been mentioned, I might wait till January as a more natural "census point").

If someone else wants to do one first, please get in contact with me so we can make it as similar to the last one as possible while also making the changes that we agreed were needed at the time.

Comment author: gwern 20 July 2012 03:35:45AM *  4 points [-]

The nice thing about this one is that it's really easy to test yourself. A plastic bag to put ice or hot water into, and some computerized mental exercise like dual n-back. I know if I thought this at anywhere close to 30% I'd test it...

EDIT: see Yvain's full version: http://squid314.livejournal.com/320770.html http://squid314.livejournal.com/321233.html http://squid314.livejournal.com/321773.html

Comment author: Yvain 27 July 2012 08:40:17PM 1 point [-]

Self-experimentation seems like a really bad way to test things about mental exhaustion. It would be way too easy to placebo myself into working for a longer amount of time without a break, when testing the condition that would support my theory. Might wait until I can find a test subject.

Comment author: Yvain 27 July 2012 07:44:52AM 11 points [-]

I have donated $1000 thanks to the nice people at Personalized Medicine.

Comment author: Wei_Dai 24 July 2012 07:33:59PM 30 points [-]

The three examples from The Art of Strategy don't seem to measure up to the book's reputation.

The Evil Plutocrat

Meanwhile, in their meeting, the Republicans think the same thing. The vote ends with all members of Congress supporting your bill, and you don't end up having to pay any money at all.

This isn't a Nash equilibrium. If I'm a congressman and I expect everyone else to vote for the bill, then I should vote against it since in that case doing so has no bad financial consequences and I get to represent my constituents.

The Hostile Takeover

Suppose the investor believes you will succeed in taking over the company. That means your competitor will not take over the company, and its $101 offer will not apply. That means that the new value of the shares will be $90, the offer you've made for the second half of shares.

It was mentioned that the stock is currently worth $100 on the open market, so why should I sell my shares to you for $90 instead of selling them on the open market? Is it assumed that there aren't legal protections for minority shareholders so whoever buys 50% of the shares can plunder the company and diminish its value for the other shareholders?

The Hostile Takeover, Part II

if the motion passes 3-2, then the two 'no' voters get no compensation and the three 'yes' voters may remain on the board and will also get a spectacular prize - to wit, our company's 51% share in your company divided up evenly among them.

Presumably the board has no power to split your shares among themselves, so what you're doing is making a promise that if the board votes that way, then you will split you shares among them. But if the board members reason backwards, they would think that if they did vote that way, you'd have no reason to actually fulfill your promise. So what the author seems to be doing is exempting the "evil villain" from backwards reasoning (or equivalently, giving him alone the power of making credible promises).

Comment author: Yvain 24 July 2012 07:46:48PM *  5 points [-]

You're right; these seem to be more parables on what happens if one side has strong ability to coordinate among itself and keep precommitments and the other side does not.

In Evil Plutocrat, assuming the party whips are very good at their jobs and can enforce their decisions, I think the Nash equilibrium (correct me if I'm wrong) is for the Democrats to make 51% of Democrats vote for the bill, and the Republicans to make 51% of Republicans vote for the bill. That makes the bill pass (and therefore neither party has their opponents get money) but still allows as many Congressmen as possible to represent their constituents. I've edited the story above to reflect this. On the other hand, if the evil plutocrat valued unanimity, I think he could get away with saying "I will pay the money to whichever party gives me less support, unless both parties give me 100% support in which case no one gets the money". This would be riskier (he might have to pay up if one Congressman defected) but in theory should be able to win him unanimity.

In Hostile Takeover, although you're right, doesn't that just pass the problem on to whoever you sell it to? At some point the shareholders either have to decide not to sell the company (thus passing up the deal to get $101 for their stock) or sell the company to one of the two bidders. If the shareholders can coordinate well enough to not take the two-tiered offer, they can coordinate well enough to just all take the $101 offer, which is superior to selling on the open market. This problem looks like what happens if the shareholders can't coordinate.

I agree that Hostile Takeover Part II only works if we assume that you have much stronger powers of precommitment than anyone on the board, and so should be understood as a parable about what happens when one party can precommit better than another. If it were important to rescue the story from this loophole, I guess we could imagine a situation where any motion passed with the consent of someone who owned shares automatically transferred those shares, and so your lackey's vote on Hostile Takeover allows those shares to be transferred automatically?

I rewrote / adapted some of these so they wouldn't be outright plagiarism, and I don't have the book with me but it's entirely possible the errors are mine and not theirs.

Comment author: ViEtArmis 24 July 2012 07:20:09PM *  0 points [-]

Even without a precommitment etc., there isn't direct incentive to be the first or second "yes" vote, only the third. If you had two shills on the board, it's a much stronger scenario.

Comment author: Yvain 24 July 2012 07:33:51PM 3 points [-]

But since there's such a strong incentive to be the third, if you are the second-most-senior-director and know that all the directors are strawmen-rational-actors, you can be pretty confident that if you vote yes, the most-senior-director will also vote yes.

Comment author: Spurlock 24 July 2012 03:42:01PM 3 points [-]

Is it rational (even straw-man rational) to enter the dollar auction after one person has already entered it? It should be obvious that you'll both happily keep bidding at least up to $20, that you have at best a 50% chance of getting the $20, and that even if you do get it you will almost certainly make a negligible amount of money even if the bidding stays under $20. So after one person has already bid, it seems like the action "enter this auction" has a clearly negative expected utility.

Comment author: Yvain 24 July 2012 07:29:54PM *  4 points [-]

If you can go through that chain of reasoning, so can the other person - therefore, it doesn't seem entirely ridiculous to me to bid $2 to the other person's $1 in the hope that they won't want to enter a bidding war and you'll win $18.

Let's say there's an X% chance you expect the other person to surrender and let you have the $20 for $2 rather than enter the bidding war, and let's also say you don't intend to ever make a bid after your first bid of $2. Then expected value is (X)(20) - (1-X)(2) = 20x - (2 - 2X) = 22X - 2. If X is greater than 1/11 or about 9%, then it's profitable to enter the auction. So unless you're greater than 91% sure that the other person will start a bidding war instead of sacrificing their $1 and letting you have the money, it's positive expected value to enter the auction.

Comment author: ViEtArmis 24 July 2012 03:51:30PM 3 points [-]

Your lackey proposes as follows: “I move that we vote upon the following: that if this motion passes unanimously, all members of the of the Board resign immediately and are given a reasonable compensation; that if this motion passes 4-1 that the Director who voted against it must retire without compensation, and the four directors who voted in favor may stay on the Board; and that if the motion passes 3-2, then the two 'no' voters get no compensation and the three 'yes' voters may remain on the board and will also get a spectacular prize - to wit, our company's 51% share in your company divided up evenly among them.”

Considering the reasoning that ends in "everyone is kicked off the board," wouldn't they all talk about it for a few minutes and then reject the proposal 4-1 (or maybe 3-2)?

Comment author: Yvain 24 July 2012 07:13:33PM *  8 points [-]

This seems much like the Prisoners' Dilemma. Yes, you can avoid it easily if you can talk beforehand and trust everyone to go through with their precommitments. If you can't talk or you don't trust what they say, then it's much harder to avoid. After all, if the first two directors cooperated with the plan by voting no, then the second two directors would have a very high incentive to defect and vote yes.

In practice people are usually able to solve these for much the same reasons they can usually solve prisoners' dilemmas - things like altruism and reputational penalties.

Game Theory As A Dark Art

50 Post author: Yvain 24 July 2012 03:27AM

One of the most charming features of game theory is the almost limitless depths of evil to which it can sink.

Your garden-variety evils act against your values. Your better class of evil, like Voldemort and the folk-tale version of Satan, use your greed to trick you into acting against your own values, then grab away the promised reward at the last moment. But even demons and dark wizards can only do this once or twice before most victims wise up and decide that taking their advice is a bad idea. Game theory can force you to betray your deepest principles for no lasting benefit again and again, and still leave you convinced that your behavior was rational.

Some of the examples in this post probably wouldn't work in reality; they're more of a reductio ad absurdum of the so-called homo economicus who acts free from any feelings of altruism or trust. But others are lifted directly from real life where seemingly intelligent people genuinely fall for them. And even the ones that don't work with real people might be valuable in modeling institutions or governments.

Of the following examples, the first three are from The Art of Strategy; the second three are relatively classic problems taken from around the Internet. A few have been mentioned in the comments here already and are reposted for people who didn't catch them the first time.



The Evil Plutocrat


You are an evil plutocrat who wants to get your pet bill - let's say a law that makes evil plutocrats tax-exempt - through the US Congress. Your usual strategy would be to bribe the Congressmen involved, but that would be pretty costly - Congressmen no longer come cheap. Assume all Congressmen act in their own financial self-interest, but that absent any financial self-interest they will grudgingly default to honestly representing their constituents, who hate your bill (and you personally). Is there any way to ensure Congress passes your bill, without spending any money on bribes at all?

Yes. Simply tell all Congressmen that if your bill fails, you will donate some stupendous amount of money to whichever party gave the greatest percent of their votes in favor.

Suppose the Democrats try to coordinate among themselves. They say “If we all oppose the bill, then if even one Republican supports the bill, the Republicans will get lots of money they can spend on campaigning against us. If only one of us supports the bill, the Republicans may anticipate this strategy and two of them may support it. The only way to ensure the Republicans don't gain a massive windfall and wipe the floor with us next election is for most of us to vote for the bill.”

Meanwhile, in their meeting, the Republicans think the same thing. The vote ends with most members of Congress supporting your bill, and you don't end up having to pay any money at all.

The Hostile Takeover

You are a ruthless businessman who wants to take over a competitor. The competitor's stock costs $100 a share, and there are 1000 shares, distributed among a hundred investors who each own ten. That means the company ought to cost $100,000, but you don't have $100,000. You only have $98,000. Worse, another competitor with $101,000 has made an offer for greater than the value of the company: they will pay $101 per share if they end up getting all of the shares. Can you still manage to take over the company?

Yes. You can make what is called a two-tiered offer. Suppose all investors get a chance to sell shares simultaneously. You will pay $105 for 500 shares - better than they could get from your competitor - but only pay $90 for the other 500. If you get fewer than 500 shares, all will sell for $105; if you get more than 500, you will start by distributing the $105 shares evenly among all investors who sold to you, and then distribute out as many of the $90 shares as necessary (leaving some $90 shares behind except when all investors sell to you) . And you will do this whether or not you succeed in taking over the company - if only one person sells you her share, then that one person gets $105.

Suppose an investor believes you're not going to succeed in taking over the company. That means you're not going to get over 50% of shares. That means the offer to buy 500 shares for $105 will still be open. That means the investor can either sell her share to you (for $105) or to your competitor (for $101). Clearly, it's in this investor's self-interest to sell to you.

Suppose the investor believes you will succeed in taking over the company. That means your competitor will not take over the company, and its $101 offer will not apply. That means that the new value of the shares will be $90, the offer you've made for the second half of shares. So they will get $90 if they don't sell to you. How much will they get if they do sell to you? They can expect half of their ten shares to go for $105 and half to go for $90; they will get a total of $97.50 per share. $97.50 is better than $90, so their incentive is to sell to you.

Suppose the investor believes you are right on the cusp of taking over the company, and her decision will determine the outcome. In that case, you have at most 499 shares. When the investor gives you her 10 shares, you will end up with 509 - 500 of which are $105 shares and 9 of which are $90 shares. If these are distributed randomly, investors can expect to make on average $104.73 per share, compared to $101 if your competitor buys the company.

Since all investors are thinking along these lines, they all choose to buy shares from you instead of your competitor. You pay out an average of $97.50 per share, and take over the company for $97,500, leaving $500 to spend on the victory party.

The stockholders, meanwhile, are left wondering why they just all sold shares for $97.50 when there was someone else who was promising them $101.

The Hostile Takeover, Part II

Your next target is a small family-owned corporation that has instituted what they consider to be invincible protection against hostile takeovers. All decisions are made by the Board of Directors, who serve for life. Although shareholders vote in the new members of the Board after one of them dies or retires, Board members can hang on for decades. And all decisions about the Board, impeachment of its members, and enforcement of its bylaws are made by the Board itself, with members voting from newest to most senior.

So you go about buying up 51% of the stock in the company, and sure enough, a Board member retires and is replaced by one of your lackeys. This lackey can propose procedural changes to the Board, but they have to be approved by majority vote. And at the moment the other four directors hate you with a vengeance, and anything you propose is likely to be defeated 4-1. You need those other four windbags out of there, and soon, but they're all young and healthy and unlikely to retire of their own accord.

The obvious next step is to start looking for a good assassin. But if you can't find one, is there any way you can propose mass forced retirement to the Board and get them to approve it by majority vote? Even better, is there any way you can get them to approve it unanimously, as a big “f#@& you” to whoever made up this stupid system?

Yes. Your lackey proposes as follows: “I move that we vote upon the following: that if this motion passes unanimously, all members of the of the Board resign immediately and are given a reasonable compensation; that if this motion passes 4-1 that the Director who voted against it must retire without compensation, and the four directors who voted in favor may stay on the Board; and that if the motion passes 3-2, then the two 'no' voters get no compensation and the three 'yes' voters may remain on the board and will also get a spectacular prize - to wit, our company's 51% share in your company divided up evenly among them.”

Your lackey then votes “yes”. The second newest director uses backward reasoning as follows:

Suppose that the vote were tied 2-2. The most senior director would prefer to vote “yes”, because then she gets to stay on the Board and gets a bunch of free stocks.

But knowing that, the second most senior director (SMSD) will also vote 'yes'. After all, when the issue reaches the SMSD, there will be one of the following cases:

1.  If there is only one yes vote (your lackey's), the SMSD stands to gain from voting yes, knowing that will produce a 2-2 tie and make the most senior director vote yes to get her spectacular compensation. This means the motion will pass 3-2, and the SMSD will also remain on the board and get spectacular compensation if she votes yes, compared to a best case scenario of remaining on the board if she votes no.

2. If there are two yes votes, the SMSD must vote yes - otherwise, it will go 2-2 to the most senior director, who will vote yes, the motion will pass 3-2, and the SMSD will be forced to retire without compensation.

3. And if there are three yes votes, then the motion has already passed, and in all cases where the second most senior director votes “no”, she is forced to retire without compensation. Therefore, the second most senior director will always vote “yes”.

Since your lackey, the most senior director, and the second most senior director will always vote "yes", we can see that the other two directors, knowing the motion will pass, must vote "yes" as well in order to get any compensation at all. Therefore, the motion passes unanimously and you take over the company at minimal cost.

The Dollar Auction

You are an economics professor who forgot to go to the ATM before leaving for work, and who has only $20 in your pocket. You have a lunch meeting at a very expensive French restaurant, but you're stuck teaching classes until lunchtime and have no way to get money. Can you trick your students into giving you enough money for lunch in exchange for your $20, without lying to them in any way?

Yes. You can use what's called an all-pay auction, in which several people bid for an item, as in a traditional auction, but everyone pays their bid regardless of whether they win or lose (in a common variant, only the top two bidders pay their bids).

Suppose one student, Alice, bids $1. This seems reasonable - paying $1 to win $20 is a pretty good deal. A second student, Bob, bids $2. Still a good deal if you can get a twenty for a tenth that amount.

The bidding keeps going higher, spurred on by the knowledge that getting a $20 for a bid of less than $20 would be pretty cool. At some point, maybe Alice has bid $18 and Bob has bid $19.

Alice thinks: “What if I raise my bid to $20? Then certainly I would win, since Bob would not pay more than $20 to get $20, but I would only break even. However, breaking even is better than what I'm doing now, since if I stay where I am Bob wins the auction and I pay $18 without getting anything.” Therefore Alice bids $20.

Bob thinks “Well, it sounds pretty silly to bid $21 for a twenty dollar bill. But if I do that and win, I only lose a dollar, as opposed to bowing out now and losing my $19 bid.” So Bob bids $21.

Alice thinks “If I give up now, I'll lose a whole dollar. I know it seems stupid to keep going, but surely Bob has the same intuition and he'll give up soon. So I'll bid $22 and just lose two dollars...”

It's easy to see that the bidding could in theory go up with no limits but the players' funds, but in practice it rarely goes above $200.

...yes, $200. Economist Max Bazerman claims that of about 180 such auctions, seven have made him more than $100 (ie $50 from both players) and his highest take was $407 (ie over $200 from both players).

In any case, you're probably set for lunch. If you're not, take another $20 from your earnings and try again until you are - the auction gains even more money from people who have seen it before than it does from naive bidders (!) Bazerman, for his part, says he's made a total of $17,000 from the exercise.

At that point you're starting to wonder why no one has tried to build a corporation around this, and unsurprisingly, the online auction site Swoopo appears to be exactly that. More surprisingly, they seem to have gone bankrupt last year, suggesting that maybe H.L. Mencken was wrong and someone has gone broke underestimating people's intelligence.

The Bloodthirsty Pirates

You are a pirate captain who has just stolen $17,000, denominated entirely in $20 bills, from a very smug-looking game theorist. By the Pirate Code, you as the captain may choose how the treasure gets distributed among your men. But your first mate, second mate, third mate, and fourth mate all want a share of the treasure, and demand on threat of mutiny the right to approve or reject any distribution you choose.You expect they'll reject anything too lopsided in your favor, which is too bad, because that was totally what you were planning on.

You remember one fact that might help you - your crew, being bloodthirsty pirates, all hate each other and actively want one another dead. Unfortunately, their greed seems to have overcome their bloodlust for the moment, and as long as there are advantages to coordinating with one another, you won't be able to turn them against their fellow sailors. Doubly unfortunately, they also actively want you dead.

You think quick. “Aye,” you tell your men with a scowl that could turn blood to ice, “ye can have yer votin' system, ye scurvy dogs” (you're that kind of pirate). “But here's the rules: I propose a distribution. Then you all vote on whether or not to take it. If a majority of you, or even half of you, vote 'yes', then that's how we distribute the treasure. But if you vote 'no', then I walk the plank to punish me for my presumption, and the first mate is the new captain. He proposes a new distribution, and again you vote on it, and if you accept then that's final, and if you reject it he walks the plank and the second mate becomes the new captain. And so on.”

Your four mates agree to this proposal. What distribution should you propose? Will it be enough to ensure your comfortable retirement in Jamaica full of rum and wenches?

Yes. Surprisingly, you can get away with proposing that you get $16,960, your first mate gets nothing, your second mate gets $20, your third mate gets nothing, and your fourth mate gets $20 - and you will still win 3 -2.

The fourth mate uses backward reasoning like so: Suppose there were only two pirates left, me and the third mate. The third mate wouldn't have to promise me anything, because if he proposed all $17,000 for himself and none for me, the vote would be 1-1 and according to the original rules a tie passes. Therefore this is a better deal than I would get if it were just me and the third mate.

But suppose there were three pirates left, me, the third mate, and the second mate. Then the second mate would be the new captain, and he could propose $16,980 for himself, $0 for the third mate, and $20 for me. If I vote no, then it reduces to the previous case in which I get nothing. Therefore, I should vote yes and get $20. Therefore, the final vote is 2-1 in favor.

But suppose there were four pirates left: me, the third mate, the second mate, and the first mate. Then the first mate would be the new captain, and he could propose $16,980 for himself, $20 for the third mate, $0 for the second mate, and $0 for me. The third mate knows that if he votes no, this reduces to the previous case, in which he gets nothing. Therefore, he should vote yes and get $20. Therefore, the final vote is 2-2, and ties pass.

(He might also propose $16980 for himself, $0 for the second mate, $0 for the third mate, and $20 for me. But since he knows I am a bloodthirsty pirate who all else being equal wants him dead, I would vote no since I could get a similar deal from the third mate and make the first mate walk the plank in the bargain. Therefore, he would offer the $20 to the third mate.)

But in fact there are five pirates left: me, the third mate, the second mate, the first mate, and the captain. The captain has proposed $16,960 for himself, $20 for the second mate, and $20 for me. If I vote no, this reduces to the previous case, in which I get nothing. Therefore, I should vote yes and get $20.

(The captain would avoid giving the $20s to the third and fourth rather than to the second and fourth mates for a similar reason to the one given in the previous example - all else being equal, the pirates would prefer to watch him die.)

The second mate thinks along the same lines and realizes that if he votes no, this reduces to the case with the first mate, in which the second mate also gets nothing. Therefore, he too votes yes.

Since you, as the captain, obviously vote yes as well, the distribution passes 3-2. You end up with $16,980, and your crew, who were so certain of their ability to threaten you into sharing the treasure, each end up with either a single $20 or nothing.

The Prisoners' Dilemma, Redux

This sequence previously mentioned the popularity of Prisoners' Dilemmas as gimmicks on TV game shows. In one program, Golden Balls, contestants do various tasks that add money to a central “pot”. By the end of the game, only two contestants are left, and are offered a Prisoners' Dilemma situation to split the pot between them. If both players choose to “Split”, the pot is divided 50-50. If one player “Splits” and the other player “Steals”, the stealer gets the entire pot. If both players choose to “Steal”, then no one gets anything. The two players are allowed to talk to each other before making a decision, but like all Prisoner's Dilemmas, the final choice is made simultaneously and in secret.

You are a contestant on this show. You are actually not all that evil - you would prefer to split the pot rather than to steal all of it for yourself - but you certainly don't want to trust the other guy to have the same preference. In fact, the other guy looks a bit greedy. You would prefer to be able to rely on the other guy's rational self-interest rather than on his altruism. Is there any tactic you can use before the choice, when you're allowed to communicate freely, in order to make it rational for him to cooperate?

Yes. In one episode of Golden Balls, a player named Nick successfully meta-games the game by transforming it from the Prisoner's Dilemma (where defection is rational) to the Ultimatum Game (where cooperation is rational)

Nick tells his opponent: “I am going to choose 'Steal' on this round.” (He then immediately pressed his button; although the show hid which button he pressed, he only needed to demonstrate that he had committed and his mind could no longer be changed) “If you also choose 'Steal', then for certain neither of us gets any money. If you choose 'Split', then I get all the money, but immediately after the game, I will give you half of it. You may not trust me on this, and that's understandable, but think it through. First, there's no less reason to think I'm trustworthy than if I had just told you I pressed 'Split' to begin with, the way everyone else on this show does. And second, now if there's any chance whatsoever that I'm trustworthy, then that's some chance of getting the money - as opposed to the zero chance you have of getting the money if you choose 'Steal'.”

Nick's evaluation is correct. His opponent can either press 'Steal', with a certainty of getting zero, or press 'Split', with a nonzero probability of getting his half of the pot depending on Nick's trustworthiness.

But this solution is not quite perfect, in that one can imagine Nick's opponent being very convinced that Nick will cheat him, and deciding he values punishing this defection more than the tiny chance that Nick will play fair. That's why I was so impressed to see cousin_it propose what I think is an even better solution on the Less Wrong thread on the matter:

This game has multiple Nash equilibria and cheap talk is allowed, so correlated equilibria are possible. Here's how you implement a correlated equilibrium if your opponent is smart enough:

"We have two minutes to talk, right? I'm going to ask you to flip a coin (visibly to both of us) at the last possible moment, the exact second where we must cease talking. If the coin comes up heads, I promise I'll cooperate, you can just go ahead and claim the whole prize. If the coin comes up tails, I promise I'll defect. Please cooperate in this case, because you have nothing to gain by defecting, and anyway the arrangement is fair, isn't it?"

This sort of clever thinking is, in my opinion, the best that game theory has to offer. It shows that game theory need not be only a tool of evil for classical figures of villainy like bloodthirsty pirate captains or corporate raiders or economists, but can also be used to create trust and ensure cooperation between parties with common interests.

Comment author: [deleted] 20 July 2012 07:33:36PM *  5 points [-]

I like Majority Judgement, which is like range voting except instead of sorting candidates by the sum of the scores each of them gets, you use the median of the scores. IIUC it's been proven that it's the system where tactical voting is hardest (for a certain definition of “hardest”).

In response to comment by [deleted] on Imperfect Voting Systems
Comment author: Yvain 21 July 2012 03:54:02AM -1 points [-]

I also like Majority Judgment.

Comment author: Yvain 20 July 2012 02:37:30AM *  2 points [-]

Irrationality game comment

The importance of waste heat in the brain is generally under-appreciated. An overheated brain is a major source of mental exhaustion, akrasia, and brain fog. One easy way to increase the amount of practical intelligence we can bring to bear on complicated tasks (with or without an accompanying increase in IQ itself) is to improving cooling in the brain. This would be most effective with some kind of surgical cooling system thingy, but even simple things like being in a cold room could help

Confidence: 30%

Comment author: CronoDAS 20 July 2012 12:16:02AM 6 points [-]

The two theorems you quoted technically don't apply to range voting...

Comment author: Yvain 20 July 2012 12:21:45AM 2 points [-]

Arrow doesn't, Gibbard-Satterthwaite still does to some degree.

Imperfect Voting Systems

34 Yvain 20 July 2012 12:07AM

Stalin once (supposedly) said that “He who casts the votes determines nothing; he who counts the votes determines everything “ But he was being insufficiently cynical. He who chooses the voting system may determine just as much as the other two players.

The Art of Strategy gives some good examples of this principle: here's an adaptation of one of them. Three managers are debating whether to give a Distinguished Employee Award to a certain worker. If the worker gets the award, she must receive one of two prizes: a $50 gift certificate, or a $10,000 bonus.

One manager loves the employee and wants her to get the $10,000; if she can't get the $10,000, she should at least get a gift certificate. A second manager acknowledges her contribution but is mostly driven by cost-cutting; she'd be happiest giving her the gift certificate, but would rather refuse to recognize her entirely than lose $10,000. And the third manager dislikes her and doesn't want to recognize her at all - but she also doesn't want the company to gain a reputation for stinginess, so if she gets recognized she'd rather give her the $10,000 than be so pathetic as to give her the cheap certificate.

The managers arrange a meeting to determine the employee's fate. If the agenda tells them to vote for or against giving her an award, and then proceed to determine the prize afterwards if she wins, then things will not go well for the employee. Why not? Because the managers reason as follows: if she gets the award, Manager 1 and Manager 3 will vote for the $10,000 prize, and Manager 2 will vote for the certificate.  Therefore, voting for her to get the award is practically the same as voting for her to get the $10,000 prize. That means Manager 1, who wants her to get the prize, will vote yes on the award, but Managers 2 and 3, who both prefer no award to the $10,000, will strategically vote not to give her the award. Result: she doesn't get recognized for her distinguished service.

But suppose the employee involved happens to be the secretary arranging the meeting where the vote will take place. She makes a seemingly trivial change to the agenda: the managers will vote for what the prize should be first, and then vote on whether to give it to her.

If the managers decide the appropriate prize is $10,000, then the motion to give the award will fail for exactly the same reasons it did above. But if the managers decide the certificate is appropriate, then Manager 1 and 2, who both prefer the certificate to nothing, will vote in favor of giving the award. So the three managers, thinking strategically, realize that the decision before them, which looks like “$10 grand or certificate”, is really “No award or certificate”. Since 1 and 2 both prefer the certificate to nothing, they vote that the certificate is the appropriate prize (even though Manager 1 doesn't really believe this) and the employee ends out with the gift certificate.

But if the secretary is really smart, she may set the agenda as follows: The managers first vote whether or not to give $10,000, and if that fails, they next vote whether or not to give the certificate; if both votes fail the employee gets nothing. Here the managers realize that if the first vote (for $10,000) fails, the next vote (certificate or nothing) will pass, since two managers prefer certificate to nothing as mentioned before. So the true choice in the first vote is “$10,000 versus certificate”. Since two managers (1 and 3) prefer the $10,000 to the certificate, those two start by voting to give the full $10,000, and this is what the employee gets.

So we see that all three options are possible outcomes, and that the true power rests not in the hands of any individual manager, but in the secretary who determines how the voting takes place.

Americans have a head start in understanding the pitfalls of voting systems thanks to the so-called two party system. Every four years, they face quandaries like "If leftists like me vote for Nader instead of Gore just because we like him better, are we going to end up electing Bush because we've split the leftist vote?"

Empirically, yes. The 60,000 Florida citizens who voted Green in 2000 didn't elect Nader. However, they did make Gore lose to Bush by a mere 500 votes. The last post discussed a Vickrey auction, a style of auction in which you have have no incentive to bid anything except your true value. Wouldn't it be nice if we had an electoral system with the same property: one where you should always vote for the candidate you actually support? If such a system existed, we would have ample reason to institute it and could rest assured that no modern-day Stalin was manipulating us via the choice of voting system we used.

Some countries do claim to have better systems than the simple winner-takes-all approach of the United States. My own adopted homeland of Ireland uses a system called “single transferable vote” (also called instant-runoff vote), in which voters rank the X candidates from 1 to X. If a candidate has the majority of first preference votes (or a number of first preference votes greater than the number of positions to fill divided by the number of candidates, in elections with multiple potential winners like legislative elections), then that candidate wins and any surplus votes go to their voters' next preference. If no one meets the quota, then the least popular candidate is eliminated and their second preference votes become first preferences. The system continues until all available seats are full.

For example, suppose I voted (1: Nader), (2: Gore), (3: Bush). The election officials tally all the votes and find that Gore has 49 million first preferences, Bush has 50 million, and Nader has 5 million. There's only one presidency, so a candidate would have to have a majority of votes (greater than 52 million out of 104 million) to win. Since no one meets that quota, the lowest ranked candidate gets eliminated - in this case, Nader. My vote now goes to my second preference, Gore. If 4 million Nader voters put Gore second versus 1 million who put Bush second, the tally's now at 53 million Gore, 51 million Bush. Gore has greater than 52 million and wins the election - the opposite result from if we'd elected a president the traditional way.

Another system called Condorcet voting also uses a list of all candidates ranked in order, but uses the information to run mock runoffs between each of them. So a Condorcet system would use the ballots to run a Gore/Nader match (which Gore would win), a Gore/Bush match (which Gore would win), and a Bush/Nader match (which Bush would win). Since Gore won all of his matches, he becomes President. This becomes complicated when no candidate wins all of his matches (imagine Gore beating Nader, Bush beating Gore, but Nader beating Bush in a sort of Presidential rock-paper-scissors.) Condorcet voting has various options to resolve this; some systems give victory to the candidate whose greatest loss was by the smallest margin, and others to candidates who defeated the greatest number of other candidates.

Do these systems avoid the strategic voting that plagues American elections? No. For example, both Single Transferable Vote and Condorcet voting sometimes provide incentives to rank a candidate with a greater chance of winning higher than a candidate you prefer - that is, the same "vote Gore instead of Nader" dilemma you get in traditional first-past-the-post.

There are many other electoral systems in use around the world, including several more with ranking of candidates, a few that do different sorts of runoffs, and even some that ask you to give a numerical rating to each candidate (for example “Nader 10, Gore 6, Bush -100000”). Some of them even manage to eliminate the temptation to rank a non-preferred candidate first. But these work only at the expense of incentivizing other strategic manuevers, like defining “approved candidate” differently or exaggerating the difference between two candidates.

So is there any voting system that automatically reflects the will of the populace in every way without encouraging tactical voting? No. Various proofs, including the Gibbard-Satterthwaite Theorem and the better-known Arrow Impossibility Theorem show that many of the criteria by which we would naturally judge voting systems are mutually incompatible and that all reasonable systems must contain at least some small element of tactics (one example of an unreasonable system that eliminates tactical voting is picking one ballot at random and determining the results based solely on its preferences; the precise text of the theorem rules out “nondeterministic or dictatorial” methods).

This means that each voting system has its own benefits and drawbacks, and that which one people use is largely a matter of preference. Some of these preferences reflect genuine concern about the differences between voting systems: for example, is it better to make sure your system always elects the Condorcet winner, even if that means the system penalizes candidates who are too similar to other candidates? Is it better to have a system where you can guarantee that participating in the election always makes your candidate more likely to win, or one where you can be sure that everyone voting exactly the opposite will never elect the same candidate?

But in practice, these preferences tend to be political and self-interested. This was recently apparent in Britain, which voted last year on a referendum to change the voting system. The Liberal Democrats, who were perpetually stuck in the same third-place situation as Nader in the States, supported a change to a form of instant runoff voting which would have made voting Lib Dem a much more palatable option; the two major parties opposed it probably for exactly that reason.

Although no single voting system is mathematically perfect, several do seem to do better on the criteria that real people care about; look over Wikipedia's section on the strengths and weaknesses of different voting systems to see which one looks best.

Comment author: [deleted] 15 July 2012 05:42:38PM 11 points [-]

Suppose an antique dealer buys a table for $50 [...] The dealer should never sell for less than $50

If the dealer doesn't inherently value the table (for more than firewood, etc.) and nobody else in the world values (or ever will value) the table for more than $40, then the dealer should sell for less than $50.

Sunk costs.

In response to comment by [deleted] on Bargaining and Auctions
Comment author: Yvain 20 July 2012 12:05:01AM 0 points [-]

On the one hand, good point. On the other, it does seem like the $50 number plays an important role in negotiations, maybe because of the generalization to a marketplace, where dealers who sell for above $50 continue to be furniture dealers and are generally satisfied with the deal, but dealers who sell for under $50 go out of business. At the very least $50 makes a powerful Schelling point.

Bargaining and Auctions

29 Yvain 15 July 2012 05:01PM

Some people have things. Other people want them. Economists agree that the eventual price will be set by supply and demand, but both parties have tragically misplaced their copies of the Big Book Of Levels Of Supply And Demand For All Goods. They're going to have to decide on a price by themselves.

When the transaction can be modeled by the interaction of one seller and one buyer, this kind of decision usually looks like bargaining. When it's best modeled as one seller and multiple buyers (or vice versa), the decision usually looks like an auction. Many buyers and many sellers produce a marketplace, but this is complicated and we'll stick to bargains and auctions for now.

Simple bargains bear some similarity to the Ultimatum Game. Suppose an antique dealer has a table she values at $50, and I go to the antique store and fall in love with it, believing it will add $400 worth of classiness to my room. The dealer should never sell for less than $50, and I should never buy for more than $400, but any value in between would benefit both of us. More specifically, it would give us a combined $350 profit. The remaining question is how to divide that $350 pot.

If I make an offer to buy at $60, I'm proposing to split the pot "$10 for you, $340 for me". If the dealer makes a counter-offer of $225, she's offering "$175 for you, $175 for me" - or an even split.

Each round of bargaining resembles the Ultimatum Game because one player proposes to split a pot, and the other player accepts or rejects. If the other player rejects the offer (for example, the dealer refuses to sell it for $60) then the deal falls through and neither of us gets any money.

But bargaining is unlike the Ultimatum Game for several reasons. First, neither player is the designated "offer-maker"; either player may begin by making an offer. Second, the game doesn't end after one round; if the dealer rejects my offer, she can make a counter-offer of her own. Third, and maybe most important, neither player is exactly sure about the size of the pot: I don't walk in knowing that the dealer bought the table for $50, and I may not really be sure I value the table at $400.

Our intuition tells us that the fairest method is to split the profits evenly at a price of $225. This number forms a useful Schelling point (remember those?) that prevents the hassle of further bargaining.

The Art of Strategy (see the beginning of Ch. 11) includes a proof that an even split is the rational choice under certain artificial assumptions. Imagine a store selling souvenirs for the 2012 Olympics. They make $1000/day each of the sixteen days the Olympics are going on. Unfortunately, the day before the Olympics, the workers decide to strike; the store will make no money without workers, and they don't have enough time to hire scabs.

Suppose Britain has some very strange labor laws that mandate the following negotiation procedure: on each odd numbered day of the Olympics, the labor union representative will approach the boss and make an offer; the boss can either accept it or reject it. On each even numbered day, the boss makes the offer to the labor union.

So if the negotiations were to drag on to the sixteenth and last day of the Olympics, on that even-numbered day the boss would approach the labor union rep. They're both the sort of straw man rationalists who would take 99-1 splits on the Ultimatum Game, so she offers the labor union rep $1 of the $1000. Since it's the last day of the Olympics and she's a straw man rationalist, the rep accepts.

But on the fifteenth day of the Olympics, the labor union rep will approach the boss. She knows that if no deal is struck today, she'll end out with $1 and the boss will end out with $999. She has to convince the boss to accept a deal on the fifteenth day instead of waiting until the sixteenth. So she offers $1 of the profits from the fifteenth day to the boss, with the labor union keeping the rest; now their totals are $1000 for the workers, $1000 for the boss. Since $1000 is better than $999, the boss agrees to these terms and the strike is ended on the fifteenth day.

We can see by this logic that on odd numbered days the boss and workers get the same amount, and on even numbered days the boss gets more than the workers but the ratio converges to 1:1 as the length of the negotiations increase. If they were negotiating an indefinite contract, then even if the boss made the first move we might expect her to offer an even split.

So both some intuitive and some mathematical arguments lead us to converge on this idea of an even split of the sort that gives us the table for $225. But if I want to be a “hard bargainer” - the kind of person who manages to get the table for less than $225 - I have a couple of things I could try.

I could deceive the seller as to how much I valued the table. This is a pretty traditional bargaining tactic: “That old piece of junk? I'd be doing you a favor for taking it off your hands.” Here I'm implicitly claiming that the dealer must have paid less than $50, and that I would get less than $400 worth of value. If the dealer paid $20 and I'd only value it to the tune of $300, then splitting the profit evenly would mean a final price of $160. The dealer could then be expected to counter my move with his own claim as to the table's value: “$160? Do I look like I was born yesterday? This table was old in the time of the Norman Conquest! Its wood comes from a tree that grows on an enchanted island in the Freptane Sea which appears for only one day every seven years!” The final price might be determined by how plausible we each considered the other's claims.

Or I could rig the Ultimatum Game. Used car dealerships are notorious for adding on “extras” after you've agreed on a price over the phone (“Well yes, we agreed the car was $5999, but if you want a steering wheel, that costs another $200.”) Somebody (possibly an LWer?) proposed showing up to the car dealership without any cash or credit cards, just a check made out for the agreed-upon amount; the dealer now has no choice but to either take the money or forget about the whole deal. In theory, I could go to the antique dealer with a check made out for $60 and he wouldn't have a lot of options (though do remember that people usually reject ultimata of below about 70-30). The classic bargaining tactic of “I am but a poor chimney sweep with only a few dollars to my name and seven small children to feed and I could never afford a price above $60” seems closely related to this strategy.

And although we're still technically talking about transactions with only one buyer and seller, the mere threat of another seller can change the balance of power drastically. Suppose I tell the dealer I know of another dealer who sells modern art for a fixed price of $300, and that the modern art would add exactly as much classiness to my room as this antique table - that is, I only want one of the two and I'm  indifferent between them. Now we're no longer talking about coming up with a price between $50 and $400 - anything over $300 and I'll reject it and go to the other guy. Now we're talking about splitting the $250 profit between $50 and $300, and if we split it evenly I should expect to pay $175.

(why not $299? After all, the dealer knows $299 is better than my other offer. Because we're still playing the Ultimatum Game, that's why. And if it was $299, then having a second option - art that I like as much as the table - would actually make my bargaining position worse - after all, I was getting it for $225 before.)

Negotiation gurus call this backup option the BATNA (“Best Alternative To Negotiated Agreement”) and consider it a useful thing to have. If only one participant in the negotiation has a BATNA greater than zero, that person is less desperate, needs the agreement less, and can hold out for a better deal - just as my $300 art allowed me to lower the asking price of the table from $225 to $175.

This “one buyer, one seller” model is artificial, but from here we can start to see how the real world existence of other buyers and sellers serve as BATNAs for both parties and how such negotiations eventually create the supply and demand of the marketplace.

The remaining case is one seller and multiple buyers (or vice versa). Here the seller's BATNA is “sell it to the other guy”, and so a successful buyer must beat the other guy's price. In practice, this takes the form of an auction (why is this different than the previous example? Partly because in the previous example, we were comparing a negotiable commodity - the table - to a fixed price commodity - the art.)

How much should you bid at an auction? In the so-called English auction (the classic auction where a crazy man stands at the front shouting “Eighty!!! Eighty!!! We have eighty!!! Do I hear eighty-five?!? Eighty-five?!? Eighty-five to the man in the straw hat!!! Do I hear ninety?!?) the answer should be pretty obvious: keep bidding infinitesmally more than the last guy until you reach your value for the product, then stop. For example, with the $400 table, keep bidding until the price approaches $400.

But what about a sealed-bid auction, where everyone hands the auctioneer their bid and the auctioneer gives the product to the highest? Or what about the so-called “Dutch auction” where the auctioneer starts high and goes lower until someone bites (“A hundred?!? Anyone for a hundred?!? No?!? Ninety-five?!? Anyone for...yes?!? Sold for ninety-five to the man in the straw hat!!!).

The rookie mistake is to bid the amount you value the product. Remember, economists define “the amount you value the product” as “the price at which you would be indifferent between having the product and just keeping the money”. If you go to an auction planning to bid your true value, you should expect to get absolutely zero benefit out of the experience. Instead, you should bid infinitesimally more than what you predict the next highest bidder will pay, as long as this is below your value.

Thus, the auction beloved by economists as perhaps the purest example of auction forms is the Vickrey, in which everyone submits a sealed bid, the highest bidder wins, and she pays the amount of the second-highest bid. This auction has a certain very elegant property, which is that here the dominant strategy is to bid your true value. Why?

Suppose you value a table at $400. If you try to game the system by bidding $350 instead of $400, you may lose out  and can at best break even. Why? Because if the highest other bid was above $400, you wouldn't win the table in either case, and your ploy profits you nothing. And if the highest other bid was between $350 and $400 (let's say $375), now you lose the table and make $0 profit, as opposed to the $25 profit you would have made if you had bid your true value of $400, won, and paid the second-highest bid of $375. And if everyone else is below $350 (let's say $300) then you would have paid $300 in either case, and again your ploy profits you nothing. Bid above your true valuation (let's say $450) and you face similar consequences: either you wouldn't have gotten the table anyway, you get the table for the same amount as before, or you get the table for a value between $400 and $450 and now you're taking a loss.

In the real world, English, Dutch, sealed-bid and Vickrey auctions all differ a little in ways like how much information they give the bidders about each other, or whether people get caught up in the excitement of bidding, or what to do when you don't really know your true valuation. But in simplified rational models, they all end at an identical price: the true valuation of the second-highest bidder.

In conclusion, the gentlemanly way to bargain is to split the difference in profits between your and your partner's best alternative to an agreement, and gentlemanly auctions tend to end at the value of the second-highest participant. Some less gentlemanly alternatives are also available and will be discussed later.

In response to comment by [deleted] on What Is Signaling, Really?
Comment author: [deleted] 11 July 2012 12:08:49AM *  58 points [-]

Upvote this comment if you didn't notice, or weren't bothered by, the use of “less” instead of “fewer”.

In response to comment by [deleted] on What Is Signaling, Really?
Comment author: Yvain 12 July 2012 10:58:59PM *  5 points [-]

I stand corrected; apparently about 20% would have been more accurate.

But to corroborate Alicorn's story, though, I kept catching myself trying to remove that error during the proofreading stage before I remembered what I was doing.

Comment author: [deleted] 12 July 2012 10:25:26PM *  1 point [-]

Your analogy is not at all applicable. According to the American Cancer Society, men should start considering getting a prostate exam at age 50. Therefore if a 23 year old male is NOT getting a prostate exam, then he is doing exactly the right thing and still has a low risk of prostate cancer.

However, adults of ANY age who are sexually active (again, excluding monogamy, if you want), SHOULD be getting tested for STDs. Therefore if a sexually active 23 year-old female is not getting tested for STDs then she is NOT behaving responsibly, and has a much higher risk of having an STD than the older female, who has had more partners, but is tested regularly.

tl;dr- Base rates

In response to comment by [deleted] on Cultural norms in choice of mate
Comment author: Yvain 12 July 2012 10:47:15PM *  7 points [-]

You're disputing a contingent aspect of the analogy that isn't related to the problem with the OKCupid article.

To establish that "older people get tested more -> older people have less undetected disease", you first have to establish that older people started out with the same base rate of disease as younger people. The OKCupid article doesn't do that, and knb is correctly calling them out on it.

I agree that younger people in fact have higher STD rates than older people, but OKCupid makes a fallacious argument for this, and it is acceptable in philosophy to criticize fallacious arguments for true conclusions.

What Is Signaling, Really?

74 Yvain 12 July 2012 05:43PM

The most commonly used introduction to signaling, promoted both by Robin Hanson and in The Art of Strategy, starts with college degrees. Suppose, there are two kinds of people, smart people and stupid people; and suppose, with wild starry-eyed optimism, that the populace is split 50-50 between them. Smart people would add enough value to a company to be worth a $100,000 salary each year, but stupid people would only be worth $40,000. And employers, no matter how hard they try to come up with silly lateral-thinking interview questions like “How many ping-pong balls could fit in the Sistine Chapel?”, can't tell the difference between them.

Now suppose a certain college course, which costs $50,000, passes all smart people but flunks half the stupid people. A strategic employer might declare a policy of hiring (for a one year job; let's keep this model simple) graduates at $100,000 and non-graduates at $40,000.

Why? Consider the thought process of a smart person when deciding whether or not to take the course. She thinks “I am smart, so if I take the course, I will certainly pass. Then I will make an extra $60,000 at this job. So my costs are $50,000, and my benefits are $60,000. Sounds like a good deal.”

The stupid person, on the other hand, thinks: “As a stupid person, if I take the course, I have a 50% chance of passing and making $60,000 extra, and a 50% chance of failing and making $0 extra. My expected benefit is $30,000, but my expected cost is $50,000. I'll stay out of school and take the $40,000 salary for non-graduates.”

...assuming that stupid people all know they're stupid, and that they're all perfectly rational experts at game theory, to name two of several dubious premises here. Yet despite its flaws, this model does give some interesting results. For example, it suggests that rational employers will base decisions upon - and rational employees enroll in - college courses, even if those courses teach nothing of any value. So an investment bank might reject someone who had no college education, even while hiring someone who studied Art History, not known for its relevance to derivative trading.

We'll return to the specific example of education later, but for now it is more important to focus on the general definition that X signals Y if X is more likely to be true when Y is true than when Y is false. Amoral self-interested agents after the $60,000 salary bonus for intelligence, whether they are smart or stupid, will always say “Yes, I'm smart” if you ask them. So saying “I am smart” is not a signal of intelligence. Having a college degree is a signal of intelligence, because a smart person is more likely to get one than a stupid person.

Life frequently throws us into situations where we want to convince other people of something. If we are employees, we want to convince bosses we are skillful, honest, and hard-working. If we run the company, we want to convince customers we have superior products. If we are on the dating scene, we want to show potential mates that we are charming, funny, wealthy, interesting, you name it.

In some of these cases, mere assertion goes a long way. If I tell my employer at a job interview that I speak fluent Spanish, I'll probably get asked to talk to a Spanish-speaker at my job, will either succeed or fail, and if I fail will have a lot of questions to answer and probably get fired - or at the very least be in more trouble than if I'd just admitted I didn't speak Spanish to begin with. Here society and its system of reputational penalties help turn mere assertion into a credible signal: asserting I speak Spanish is costlier if I don't speak Spanish than if I do, and so is believable.

In other cases, mere assertion doesn't work. If I'm at a seedy bar looking for a one-night stand, I can tell a girl I'm totally a multimillionaire and feel relatively sure I won't be found out until after that one night - and so in this she would be naive to believe me, unless I did something only a real multimillionaire could, like give her an expensive diamond necklace.

How expensive a diamond necklace, exactly?  To absolutely prove I am a millionaire, only a million dollars worth of diamonds will do; $10,000 worth of diamonds could in theory come from anyone with at least $10,000. But in practice, people only care so much about impressing a girl at a seedy bar; if everyone cares about the same amount, the amount they'll spend on the signal depends mostly on their marginal utility of money, which in turn depends mostly on how much they have. Both a millionaire and a tenthousandaire can afford to buy $10,000 worth of diamonds, but only the millionaire can afford to buy $10,000 worth of diamonds on a whim. If in general people are only willing to spend 1% of their money on an impulse gift, then $10,000 is sufficient evidence that I am a millionaire.

But when the stakes are high, signals can get prohibitively costly. If a dozen millionaires are wooing Helen of Troy, the most beautiful woman in the world, and willing to spend arbitrarily much money on her - and if they all believe Helen will choose the richest among them - then if I only spend $10,000 on her I'll be outshone by a millionaire who spends the full million. Thus, if I want any chance with her at all, then even if I am genuinely the richest man around I might have to squander my entire fortune on diamonds.

This raises an important point: signaling can be really horrible. What if none of us are entirely sure how much Helen's other suitors have? It might be rational for all of us to spend everything we have on diamonds for her. Then twelve millionaires lose their fortunes, eleven of them for nothing. And this isn't some kind of wealth transfer - for all we know, Helen might not even like diamonds; maybe she locks them in her jewelry box after the wedding and never thinks about them again. It's about as economically productive as digging a big hole and throwing money into it.

If all twelve millionaires could get together beforehand and compare their wealth, and agree that only the wealthiest one would woo Helen, then they could all save their fortunes and the result would be exactly the same: Helen marries the wealthiest. If all twelve millionaires are remarkably trustworthy, maybe they can pull it off. But if any of them believe the others might lie about their wealth, or that one of the poorer men might covertly break their pact and woo Helen with gifts, then they've got to go through with the whole awful “everyone wastes everything they have on shiny rocks” ordeal.

Examples of destructive signaling are not limited to hypotheticals. Even if one does not believe Jared Diamond's hypothesis that Easter Island civilization collapsed after chieftains expended all of their resources trying to out-signal each other by building larger and larger stone heads, one can look at Nikolai Roussanov's study on how the dynamics of signaling games in US minority communities encourage conspicuous consumption and prevent members of those communities from investing in education and other important goods.

The Art of Strategy even advances the surprising hypothesis that corporate advertising can be a form of signaling. When a company advertises during the Super Bowl or some other high-visibility event, it costs a lot of money. To be able to afford the commercial, the company must be pretty wealthy; which in turn means it probably sells popular products and isn't going to collapse and leave its customers in the lurch. And to want to afford the commercial, the company must be pretty confident in its product: advertising that you should shop at Wal-Mart is more profitable if you shop at Wal-Mart, love it, and keep coming back than if you're likely to go to Wal-Mart, hate it, and leave without buying anything. This signaling, too, can become destructive: if every other company in your industry is buying Super Bowl commercials, then none of them have a comparative advantage and they're in exactly the same relative position as if none of them bought Super Bowl commercials - throwing money away just as in the diamond example.

Most of us cannot afford a Super Bowl commercial or a diamond necklace, and less people may build giant stone heads than during Easter Island's golden age, but a surprising amount of everyday life can be explained by signaling. For example, why did about 50% of readers get a mental flinch and an overpowering urge to correct me when I used “less” instead of “fewer” in the sentence above? According to Paul Fussell's “Guide Through The American Class System” (ht SIAI mailing list), nitpicky attention to good grammar, even when a sentence is perfectly clear without it, can be a way to signal education, and hence intelligence and probably social class. I would not dare to summarize Fussell's guide here, but it shattered my illusion that I mostly avoid thinking about class signals, and instead convinced me that pretty much everything I do from waking up in the morning to going to bed at night is a class signal. On flowers:

Anyone imagining that just any sort of flowers can be presented in the front of a house without status jeopardy would be wrong. Upper-middle-class flowers are rhododendrons, tiger lilies, amaryllis, columbine, clematis, and roses, except for bright-red ones. One way to learn which flowers are vulgar is to notice the varieties favored on Sunday-morning TV religious programs like Rex Humbard's or Robert Schuller's. There you will see primarily geraniums (red are lower than pink), poinsettias, and chrysanthemums, and you will know instantly, without even attending to the quality of the discourse, that you are looking at a high-prole setup. Other prole flowers include anything too vividly red, like red tulips. Declassed also are phlox, zinnias, salvia, gladioli, begonias, dahlias, fuchsias, and petunias. Members of the middle class will sometimes hope to mitigate the vulgarity of bright-red flowers by planting them in a rotting wheelbarrow or rowboat displayed on the front lawn, but seldom with success.

Seriously, read the essay.

In conclusion, a signal is a method of conveying information among not-necessarily-trustworthy parties by performing an action which is more likely or less costly if the information is true than if it is not true. Because signals are often costly, they can sometimes lead to a depressing waste of resources, but in other cases they may be the only way to believably convey important information.

Comment author: dlthomas 10 July 2012 09:00:42AM 4 points [-]

What are the costs associated with flowers?

Comment author: Yvain 10 July 2012 11:43:10PM 7 points [-]

I don't know enough about gardening to have a reasonable opinion on that, but here are some possibilities:

  • It takes resources in the form of studying fashion, or hiring someone else to do it for you, in order to know which flowers are in, and they're not the ones you expect. Compare to fashion in clothing; I'm probably unfashionable, and this correctly signals that I don't have enough time to keep up with trends or enough hip friends to advise me on them.

  • The high-status flowers are harder/more expensive to grow than the low status flowers. Compare to his discussion on gravel driveways being higher-status than concrete because they require more maintenance.

  • The high-status flowers are considered ugly to most untrained people, so they're a net loss unless you know you're associating with people who have been trained in taste. Compare to modern academic music, which will sound unpleasant to an untrained ear; therefore if you like it it signals not only that you yourself are trained, but that you expect the people who judge you based on your musical tastes to be trained. Hence the "any flower too vividly red is prole" comment.

  • The high-status flowers would look pretentious to low-status people (I realize this is partly an explanatory regress). For example, anyone can give their kid a name that sounds upper-class, but if the kid is poor they risk looking stupid.

Comment author: shminux 10 July 2012 06:21:34AM 4 points [-]

So, what are you signalling with this post?

Comment author: Yvain 10 July 2012 11:28:52PM 19 points [-]

Oh, the post was just a means to an end; it's the karma I'll use for signaling.

Comment author: JohnD 10 July 2012 11:25:28AM 3 points [-]

I'm curious, was the Art History comment a dig at Michael Lewis?

Comment author: Yvain 10 July 2012 11:24:09PM 3 points [-]

I didn't even know about Michael Lewis until I just looked him up. I could have used "underwater basket weaving", but I wasn't sure everyone would get the reference.

Comment author: private_messaging 07 July 2012 06:36:06PM *  2 points [-]

Originally made this account to message some people privately.

Can you explain why the first thing to update after the Galton's amazing study into imagination, was your opinion on women in general as determined by PUA's opinion on women vs women opinion on women (the balance of conflicting opinions)? Also, btw, it is in itself a great example of biased cognition: you run across some fact, and you update selectively; the fact should lower your weight for anyone's evaluation of anyone, but instead it just lowers the weight for women's evaluation of women.

Also, while I am sure that you did not consciously add it just for LW audience, if you were writing for a more general audience it does seem reasonable to assume - given that you are generally a good writer - that you would not include this sort of 'example' of application of the findings of Galton.

Comment author: Yvain 08 July 2012 04:21:12PM 1 point [-]

Replied in accordance with your username to prevent this from becoming an Endless Back-and-Forth Internet Argument Thread.

Comment author: bentarm 06 July 2012 12:55:09PM *  7 points [-]

Steve Landsburg has an interesting point about versions of the Dictator Game (and several other similar games) in which people have the option to "destroy" some or all of the money if they don't like the offer. In particular, he recently commented on the so-called "Destructor Game" ( pdf of paper here).

In this game, participants were given the option of deciding whether or not to take away some of the money that the experimenter had given to some of the other participants. When they chose to do so, the experimenters concluded that they were indulging a taste for destruction. As Landsburg sensibly points out, nothing was actually destroyed in any of these experiments. Money was simply transferred from an anonymous subject back to the experimenter.

The reason I bring this up here is because it seems like as good as place as any to get an answer to the next question - has anyone actually ever done such an experiment in which the goods to be destroyed were actually destroyed. (You can imagine giving everyone candy bars, and giving the participants the option to take someone else's candy bars and throw them into some stinking garbage heap). My instinct is that Landsburg is right, and people would be less likely to engage in destructive behaviour if they were destroying actual goods instead of just paper money, but I would be interested to see if this has ever actually been studied. Does anyone know?

Comment author: Yvain 07 July 2012 05:48:08PM *  5 points [-]

I've never heard of a "destroy the money" experiment, but the fact that most economists before Landsburg didn't think of this, and I didn't think of it, and my sources didn't think of it, makes me skeptical that the average participant in the Dictator Game is thinking of it.

I'm also reminded of stories about medical malpractice lawsuits, where juries will sometimes award really big sums of money even when they're not sure whether the doctor was guilty, on the grounds that hospitals/clinics/insurances are large faceless institutions and probably have so much money they won't miss a little. I would expect players to treat the researcher (presumably working off grant money from a big research university) the same way juries treat hospitals.

Comment author: johnswentworth 06 July 2012 11:11:48PM 1 point [-]

I don't think the "spoil the setup" experiment distinguishes TDT from the belief in fairness. Just because the second person's decision comes after the first doesn't mean it has no effect on the first. It's very much like Newcomb's problem in that regard, and one of the main points of TDT was to account for that effect. Depending on the details of the rewards and how strongly you think the other player's decisions correlate with your own, it may make sense to precommit to cooperation even if you're told the other person's choice. And if it makes sense to precommit to cooperation, that's what TDT will do (unless I'm missing something).

Comment author: Yvain 07 July 2012 05:46:57PM 0 points [-]

Do you have an alternate explanation for why so many fewer people cooperated in the "spoil the setup" experiment than in ordinary experiments?

Comment author: andreacasalotti 07 July 2012 11:37:32AM 6 points [-]

Regarding the Dana, Cain and Dawes experiment, the abstract says: "Over two studies, we found that about one third of participants were willing to exit a $10 dictator game and take $9 instead. " One third is less than "the majority of participants" stated by you. A fisherman's tale?

Comment author: Yvain 07 July 2012 05:45:57PM 1 point [-]

Good catch. I got the numbers out of Art of Strategy, then searched for the study online. Either it's a slightly different study than the one cited in the book with slightly different results, or I'm transmitting an error from there.

Comment author: cousin_it 05 July 2012 04:56:50PM *  7 points [-]

Good point, I agree with RobertLumley and you that doing good work on LW is better than doing nothing. But if you're already doing good work by LW standards (and you are!), it probably makes sense to move beyond.

Comment author: Yvain 07 July 2012 05:28:38PM *  7 points [-]

Thank you for the compliments. I don't know much about mathematics, but if you've really proven a new insight into Godel's theorem that sounds very impressive and you should certainly try to get it published formally. But I'm not sure what you're suggesting I should do. I mean, I've been talking to Luke about maybe helping with some SIAI literature, but that's probably in the same category as here.

My career plan is medicine. I blog as a hobby. One day I would like to write about medicine, but I need a job and some experience before I can do that credibly. If you or anyone else have anything else specific and interesting/important you think I should be (or even could be) doing, please let me know. Bonus points if it involves getting paid.

Comment author: Yvain 07 July 2012 04:04:49AM *  19 points [-]

As Luke helpfully taught us, negative reinforcement doesn't seem to work as well as positive. Spanking, in particular, is right out.

I've seen this claim all over the comments to Luke's post, and I don't see Luke as asserting it (or being able to). In fact, there are only two things in Luke's post that even slightly resemble this claim:

First is a warning against punishing people in a way such that the punishment could negatively incentivize the effort instead of the failure. This is a good point, but doesn't apply in a lot of cases - for example, punishing a child for hitting a sibling.

Second is the following quote:

When trying to maintain order in a class, ignore unruly behavior and praise good behavior (Madsen et al. 1968; McNamara 1987).

So I looked up McNamara 1987. It's a book chapter that reports a few studies on students getting better when praised, then says they're very weak, no one's been able to replicate them, and then goes into how a better study McNamara clearly finds more credible shows that students do worse with the sort of hokey planned praise used in the previous studies.

Then I looked up Madsen et all 1968. It's a study on three problem children, Cliff, Frank and Stan. The relevant conditions were Rules, where the teachers wrote some rules on the blackboard and made the children repeat them a bunch of times, and Praise, in which the teachers praised the students for following the rules. There was no punishment condition. Writing the rules on the blackboard didn't help much, but Cliff and Frank seemed to get better after being praised a lot (Stan was equivocal).

Poor Cliff and Frank are not a sufficient basis on which to assert incredibly broad psychological theories about how you should always reward and never punish all humans in all cases.

In the case of positive behavior/reinforcement, there's nothing particularly problematic about this, but for the negative case, you're also punishing being caught/noticed/seen, which can be problematic.

This is a kind-of-sounds-good argument, but one could make equally kind-of-sounds-good arguments on the other side. For example, rewards decrease intrisic motivation by replacing them with extrinsic motivation. Or anything Alfie Kohn ever said (I'm not necessarily endorsing him, just holding him up as an example of arguments that rewards too have their problems).

Finally, I am very suspicious of the studies in this area. Not only is developmental psych generally a minefield, but "rewards work better than punishments" is so in tune with modern sensibilities that I would be really surprised if the average researcher doesn't go out gunning for that result and quietly file anything that shows punishments are better than rewards into a file drawer never to see the light of day.

(one counterexample is a recent study that shows spanking children generally gives them better life outcomes. I've seen a zillion studies that say the opposite, and I consider the entire project of observational studies of spanking totally useless because of the confounding effects of parental choice - ie parents who decide to spank or not spank their children are probably different in many other correlated ways which are much more important - but the study exists and I don't consider the spanking field nearly as settled as you do)

To speak to goal 1 first, Bryan Caplan claims flat outcomes for goal #1 under commonly tried parenting interventions, which seems counter-intuitive. More explanation of what exactly the studies in question proved would be welcome.

I've never read Caplan, but I'm working my way through The Nurture Assumption which is probably what Caplan's working off of. Nurture Assumption claims pretty much everything is flat for pretty much all kinds of parenting and we need to drop the assumption (hence the name) that parenting matters much beyond just not starving or abusing your kid. The books presents lots of complicated evidence but the strongest is adoption studies, where children raised together aren't much more similar than children raised apart (with degree of genetic relatedness held constant). Right now I believe her about 60%.

Interlude for Behavioral Economics

49 Post author: Yvain 06 July 2012 08:12PM

The so-called “rational” solutions to the Prisoners' Dilemma and Ultimatum Game are suboptimal to say the least. Humans have various kludges added by both nature or nurture to do better, but they're not perfect and they're certainly not simple. They leave entirely open the question of what real people will actually do in these situations, a question which can only be addressed by hard data.

As in so many other areas, our most important information comes from reality television. The Art of Strategy discusses a US game show “Friend or Foe” where a team of two contestants earned money by answering trivia questions. At the end of the show, the team used a sort-of Prisoner's Dilemma to split their winnings: each team member chose “Friend” (cooperate) or “Foe” (defect). If one player cooperated and the other defected, the defector kept 100% of the pot. If both cooperated, each kept 50%. And if both defected, neither kept anything (this is a significant difference from the standard dilemma, where a player is a little better off defecting than cooperating if her opponent defects).

Players chose “Friend” about 45% of the time. Significantly, this number remained constant despite the size of the pot: they were no more likely to cooperate when splitting small amounts of money than large.

Players seemed to want to play “Friend” if and only if they expected their opponents to do so. This is not rational, but it accords with the “Tit-for-Tat” strategy hypothesized to be the evolutionary solution to Prisoner's Dilemma. This played out on the show in a surprising way: players' choices started off random, but as the show went on and contestants began participating who had seen previous episodes, they began to base their decision on observable characteristics about their opponents. For example, in the first season women cooperated more often than men, so by the second season a player was cooperating more often if their opponent was a woman - whether or not that player was a man or woman themselves.

Among the superficial characteristics used, the only one to reach statistical significance according to the study was age: players below the median age of 27 played “Foe” more often than those over it (65% vs. 39%, p < .001). Other nonsignificant tendencies were for men to defect more than women (53% vs. 46%, p=.34) and for black people to defect more than white people (58% vs. 48%, p=.33). These nonsignificant tendencies became important because the players themselves attributed significance to them: for example, by the second season women were playing “Foe” 60% of the time against men but only 45% of the time against women (p<.01) presumably because women were perceived to be more likely to play “Friend” back; also during the second season, white people would play “Foe” 75% against black people, but only 54% of the time against other white people.

(This risks self-fulfilling prophecies. If I am a black man playing a white woman, I expect she will expect me to play “Foe” against her, and she will “reciprocate” by playing “Foe” herself. Therefore, I may choose to “reciprocate” against her by playing “Foe” myself, even if I wasn't originally intending to do so, and other white women might observe this, thus creating a vicious cycle.)

In any case, these attempts at coordinated play worked, but only imperfectly. By the second season, 57% of pairs chose the same option - either (C, C) or (D, D).

Art of Strategy included another great Prisoner's Dilemma experiment. In this one, the experimenters spoiled the game: they told both players that they would be deciding simultaneously, but in fact, they let Player 1 decide first, and then secretly approached Player 2 and told her Player 1's decision, letting Player 2 consider this information when making her own choice.

Why should this be interesting? From the previous data, we know that humans play “tit-for-expected-tat”: they will generally cooperate if they believe their opponent will cooperate too. We can come up with two hypotheses to explain this behavior. First, this could be a folk version of Timeless Decision Theory or Hofstadter's superrationality; a belief that their own decision literally determines their opponent's decision. Second, it could be based on a belief in fairness: if I think my opponent cooperated, it's only decent that I do the same.

The “researchers spoil the setup” experiment can distinguish between these two hypotheses. If people believe their choice determines that of their opponent, then once they know their opponent's choice they no longer have to worry and can freely defect to maximize their own winnings. But if people want to cooperate to reward their opponent, then learning that their opponent cooperated for sure should only increase their willingness to reciprocate.

The results: If you tell the second player that the first player defected, 3% still cooperate (apparently 3% of people are Jesus). If you tell the second player that the first player cooperated.........only 16%  cooperate. When the same researchers in the same lab didn't tell the second player anything, 37% cooperated.

This is a pretty resounding victory for the “folk version of superrationality” hypothesis. 21% of people wouldn't cooperate if they heard their opponent defected, wouldn't cooperate if they heard their opponent cooperated, but will cooperate if they don't know which of those two their opponent played.

Moving on to the Ultimatum Game: very broadly, the first player usually offers between 30 and 50 percent, and the second player tends to accept. If the first player offers less than about 20 percent, the second player tends to reject it.

Like the Prisoner's Dilemma, the amount of money at stake doesn't seem to matter. This is really surprising! Imagine you played an Ultimatum Game for a billion dollars. The first player proposes $990 million for herself, $10 million for you. On the one hand, this is a 99-1 split, just as unfair as $99 versus $1. On the other hand, ten million dollars!

Although tycoons have yet to donate a billion dollars to use for Ultimatum Game experiments, researchers have done the next best thing and flown out to Third World countries where even $100 can be an impressive amount of money. In games in Indonesia played for a pot containing a sixth of Indonesians' average yearly income, Indonesians still rejected unfair offers. In fact, at these levels the first player tended to propose fairer deals than at lower stakes - maybe because it would be a disaster if her offer get rejected.

It was originally believed that results in the Ultimatum Game were mostly independent of culture.  Groups in the US, Israel, Japan, Eastern Europe, and Indonesia all got more or less the same results. But this elegant simplicity was, like so many other things, ruined by the Machiguenga Indians of eastern Peru. They tend to make offers around 25%, and will accept pretty much anything.

One more interesting finding: people who accept low offers in the Ultimatum Game have lower testosterone than those who reject them.

There is a certain degenerate form of the Ultimatum Game called the Dictator Game. In the Dictator Game, the second player doesn't have the option of vetoing the first player's distribution. In fact, the second player doesn't do anything at all; the first player distributes the money, both players receive the amount of money the first player decided upon, and the game ends. A perfectly selfish first player would take 100% of the money in the Dictator Game, leaving the second player with nothing.

In a metaanalysis of 129 papers consisting of over 41,000 individual games, the average amount the first player gave the second player was 28.35%. 36% of first players take everything, 17% divide the pot equally, and 5% give everything to the second player, nearly doubling our previous estimate of what percent of people are Jesus.

The meta-analysis checks many different results, most of which are insignificant, but a few stand out. Subjects playing the dictator game “against” a charity are much more generous; up to a quarter give everything. When the experimenter promises to “match” each dollar given away (eg the dictator gets $100, but if she gives it to the second player the second player gets $200), the dictator gives much more (somewhat surprising, as this might be an excuse to keep $66 for yourself and get away with it by claiming that both players still got equal money). On the other hand, if the experimenters give the second player a free $100, so that they start off richer than the dictator, the dictator compensates by not giving them nearly as much money.

Old people give more than young people, and non-students give more than students. People from “primitive” societies give more than people from more developed societies, and the more primitive the society, the stronger the effect.  The most important factor, though? As always, sex. Women both give more and get more in dictator games.

It is somewhat inspiring that so many people give so much in this game, but before we become too excited about the fundamental goodness of humanity, Art of Strategy mentions a great experiment by Dana, Cain, and Dawes. The subjects were offered a choice: either play the Dictator Game with a second player for $10, or get $9 and the second subject is sent home and never even knows what the experiment is about. A third of participants took the second option.

So generosity in the Dictator Game isn't always about wanting to help other people. It seems to be about knowing, deep down, that some anonymous person who probably doesn't even know your name and who will never see you again is disappointed in you. Remove the little problem of the other person knowing what you did, and they will not only keep the money, but even be willing to pay the experiment a dollar to keep them quiet.

Comment author: beoShaffer 06 July 2012 03:19:56AM 0 points [-]

As with the previous entries in the sequence, I like the article but strongly suggest that you add links between sequence entries.

Comment author: Yvain 06 July 2012 03:33:08AM 2 points [-]

Thanks. I'll do that when I'm done with the whole thing, so that I don't have to keep going back and adding new "Next In Sequence" posts when I post new articles.

Comment author: AlexMennen 06 July 2012 02:59:46AM 2 points [-]

by the second season women were playing “Foe” 60% of the time against women but only 45% of the time against men (p<.01) presumably because women were perceived to be more likely to play “Friend” back

This looks backwards.

Comment author: Yvain 06 July 2012 03:06:54AM 0 points [-]

Thanks, fixed.

Comment author: Yvain 05 July 2012 02:36:54AM 9 points [-]

This was my favorite post on your blog and I'm glad you posted it here.

Comment author: Andreas_Giger 03 July 2012 12:10:36PM 0 points [-]

Most people, when they hear the rational arguments in favor of defecting every single time on the iterated 100-crime Prisoner's Dilemma, will feel some kind of emotional resistance. Thoughts like “Well, maybe I'll try cooperating anyway a few times, see if it works”, or “If I promised to cooperate with my opponent, then it would be dishonorable for me to defect on the last turn, even if it helps me out., or even “Bob is my friend! Think of all the good times we've had together, robbing banks and running straight into waiting police cordons. I could never betray him!”

I don't want to start a discussion here, I just want to clarify your position on this: Are you implying that it is perfectly rational to always assume that your opponent is a perfect rationalist?

Comment author: Yvain 04 July 2012 03:41:11PM 3 points [-]

No.

Comment author: Grognor 03 July 2012 02:14:49PM *  16 points [-]

There are many problems here.

At the end of paragraph 2 and the other examples, you say

This exactly mirrors the Prisoner's Dilemma.

But it doesn't, as you point out later in the post, because the payoff matrix isn't D-C > C-C > D-D, as you explain, but rather C-C > D-C > C-D, because of reputational effects, which is not a prisoner's dilemma. "Prisoner's dilemma" is a very specific term, and you are inflating it.

evolution is also strongly motivated [...] evolution will certainly take note.

I doubt that quite strongly!

The evolutionarily dominant strategy is commonly called “Tit-for-tat” - basically, cooperate if and only if you expect your opponent to do so.

That is not tit-for-tat! Tit-for-tat is start with cooperate and then parrot the opponent's previous move. It does not do what it "expects" the opponent to do. Furthermore, if you categorically expect your opponent to cooperate, you should defect (just like you should if you expect him to defect). You only cooperate if you expect your opponent to cooperate if he expects you to cooperate ad nauseum.

This so-called "superrationality” appears even more [...]

That is not superrationality! Superrationality achieves cooperation by reasoning that you and your opponent will get the same result for the same reasons, so you should cooperate in order to logically bind your result to C-C (since C-C and D-D are the only two options). What is with all this misuse of terminology? You write like the agents in the examples of this game are using causal decision theory (which defects all the time no matter what) and then bring up elements that cannot possibly be implemented in causal decision theory, and it grinds my gears!

And if two people with these sorts of emotional hangups play the Prisoner's Dilemma together, they'll end up cooperating on all hundred crimes, getting out of jail in a mere century and leaving rational utility maximizers to sit back and wonder how they did it.

This is in direct violation of one of the themes of Less Wrong. If "rational expected utility maximizers" are doing worse than "irrational emotional hangups", then you're using a wrong definition of "rational". You do this throughout the post, and it's especially jarring because you are or were one of the best writers for this website.

playing as a "rational economic agent" gets you a bad result

9_9

[...] anger makes us irrational. But this is the good kind of irrationality [...]

"The good kind of irrationality" is like "the good kind of bad thing". An oxymoron, by definition.

[...] if we're playing an Ultimatum Game against a human, and that human precommits to rejecting any offer less than 50-50, we're much more likely to believe her than if we were playing against a rational utility-maximizing agent

Bullshit. A rational agent is going to do what works. We know this because we stipulated that it was rational. If you mean to say a "stupid number crunching robot that misses obvious details like how to play ultimatum games" then sure it might do as you describe. But don't call it "rational".

It is distasteful and a little bit contradictory to the spirit of rationality to believe it should lose out so badly to simple emotion, and the problem might be correctable.

You think?

Downvoted.

Comment author: Yvain 04 July 2012 03:39:38PM 3 points [-]

You'll notice I used scare quotes around most of the words you objected to. I'm trying to point out the apparent paradox, using the language that game theorists and other people not already on this website would use, without claiming that the paradox is real or unsolvable.

Comment author: Yvain 04 July 2012 03:41:15AM 11 points [-]

On my settings (some automatic zooming because I have bad eyesight), the photo takes up the entire front page and I didn't realize there was anything else beneath it to scroll down to until I saw it mentioned here.

Also, I get that you're all social and popular and people-oriented, but it just seems weird to have so many random pictures of attractive people enjoying themselves but not obviously doing anything related to rationality. Maybe removing the duplicate photos would help.

Real World Solutions to Prisoners' Dilemmas

31 Yvain 03 July 2012 03:25AM

Why should there be real world solutions to Prisoners' Dilemmas? Because such dilemmas are a real-world problem.

If I am assigned to work on a school project with a group, I can either cooperate (work hard on the project) or defect (slack off while reaping the rewards of everyone else's hard work). If everyone defects, the project doesn't get done and we all fail - a bad outcome for everyone. If I defect but you cooperate, then I get to spend all day on the beach and still get a good grade - the best outcome for me, the worst for you. And if we all cooperate, then it's long hours in the library but at least we pass the class - a “good enough” outcome, though not quite as good as me defecting against everyone else's cooperation. This exactly mirrors the Prisoner's Dilemma.

Diplomacy - both the concept and the board game - involves Prisoners' Dilemmas. Suppose Ribbentrop of Germany and Molotov of Russia agree to a peace treaty that demilitarizes their mutual border. If both cooperate, they can move their forces to other theaters, and have moderate success there - a good enough outcome. If Russia cooperates but Germany defects, it can launch a surprise attack on an undefended Russian border and enjoy spectacular success there (for a while, at least!) - the best outcome for Germany and the worst for Russia. But if both defect, then neither has any advantage at the German-Russian border, and they lose the use of those troops in other theaters as well - a bad outcome for both. Again, the Prisoner's Dilemma.

Civilization - again, both the concept and the game - involves Prisoners' Dilemmas. If everyone follows the rules and creates a stable society (cooperates), we all do pretty well. If everyone else works hard and I turn barbarian and pillage you (defect), then I get all of your stuff without having to work for it and you get nothing - the best solution for me, the worst for you. If everyone becomes a barbarian, there's nothing to steal and we all lose out. Prisoner's Dilemma.

If everyone who worries about global warming cooperates in cutting emissions, climate change is averted and everyone is moderately happy. If everyone else cooperates in cutting emissions, but one country defects, climate change is still mostly averted, and the defector is at a significant economic advantage. If everyone defects and keeps polluting, the climate changes and everyone loses out. Again a Prisoner's Dilemma,

Prisoners' Dilemmas even come up in nature. In baboon tribes, when a female is in “heat”, males often compete for the chance to woo her. The most successful males are those who can get a friend to help fight off the other monkeys, and who then helps that friend find his own monkey loving. But these monkeys are tempted to take their friend's female as well. Two males who cooperate each seduce one female. If one cooperates and the other defects, he has a good chance at both females. But if the two can't cooperate at all, then they will be beaten off by other monkey alliances and won't get to have sex with anyone. Still a Prisoner's Dilemma!

So one might expect the real world to have produced some practical solutions to Prisoners' Dilemmas.

One of the best known such systems is called “society”. You may have heard of it. It boasts a series of norms, laws, and authority figures who will punish you when those norms and laws are broken.

Imagine that the two criminals in the original example were part of a criminal society - let's say the Mafia. The Godfather makes Alice and Bob an offer they can't refuse: turn against one another, and they will end up “sleeping with the fishes” (this concludes my knowledge of the Mafia). Now the incentives are changed: defecting against a cooperator doesn't mean walking free, it means getting murdered.





Both prisoners cooperate, and amazingly the threat of murder ends up making them both better off (this is also the gist of some of the strongest arguments against libertarianism: in Prisoner's Dilemmas, threatening force against rational agents can increase the utility of all of them!)

Even when there is no godfather, society binds people by concern about their “reputation”. If Bob got a reputation as a snitch, he might never be able to work as a criminal again. If a student gets a reputation for slacking off on projects, she might get ostracized on the playground. If a country gets a reputation for backstabbing, others might refuse to make treaties with them. If a person gets a reputation as a bandit, she might incur the hostility of those around her. If a country gets a reputation for not doing enough to fight global warming, it might...well, no one ever said it was a perfect system.

Aside from humans in society, evolution is also strongly motivated to develop a solution to the Prisoner's Dilemma. The Dilemma troubles not only lovestruck baboons, but ants, minnows, bats, and even viruses. Here the payoff is denominated not in years of jail time, nor in dollars, but in reproductive fitness and number of potential offspring - so evolution will certainly take note.

Most people, when they hear the rational arguments in favor of defecting every single time on the iterated 100-crime Prisoner's Dilemma, will feel some kind of emotional resistance. Thoughts like “Well, maybe I'll try cooperating anyway a few times, see if it works”, or “If I promised to cooperate with my opponent, then it would be dishonorable for me to defect on the last turn, even if it helps me out., or even “Bob is my friend! Think of all the good times we've had together, robbing banks and running straight into waiting police cordons. I could never betray him!”

And if two people with these sorts of emotional hangups play the Prisoner's Dilemma together, they'll end up cooperating on all hundred crimes, getting out of jail in a mere century and leaving rational utility maximizers to sit back and wonder how they did it.

Here's how: imagine you are a supervillain designing a robotic criminal (who's that go-to supervillain Kaj always uses for situations like this? Dr. Zany? Okay, let's say you're him). You expect to build several copies of this robot to work as a team, and expect they might end up playing the Prisoner's Dilemma against each other. You want them out of jail as fast as possible so they can get back to furthering your nefarious plots. So rather than have them bumble through the whole rational utility maximizing thing, you just insert an extra line of code: “in a Prisoner's Dilemma, always cooperate with other robots”. Problem solved.

Evolution followed the same strategy (no it didn't; this is a massive oversimplification). The emotions we feel around friendship, trust, altruism, and betrayal are partly a built-in hack to succeed in cooperating on Prisoner's Dilemmas where a rational utility-maximizer would defect a hundred times and fail miserably. The evolutionarily dominant strategy is commonly called “Tit-for-tat” - basically, cooperate if and only if your opponent did so last time.

This so-called "superrationality” appears even more clearly in the Ultimatum Game. Two players are given $100 to distribute among themselves in the following way: the first player proposes a distribution (for example, “Fifty for me, fifty for you”) and then the second player either accepts or rejects the distribution. If the second player accepts, the players get the money in that particular ratio. If the second player refuses, no one gets any money at all.

The first player's reasoning goes like this: “If I propose $99 for myself and $1 for my opponent, that means I get a lot of money and my opponent still has to accept. After all, she prefers $1 to $0, which is what she'll get if she refuses.

In the Prisoner's Dilemma, when players were able to communicate beforehand they could settle upon a winning strategy of precommiting to reciprocate: to take an action beneficial to their opponent if and only if their opponent took an action beneficial to them. Here, the second player should consider the same strategy: precommit to an ultimatum (hence the name) that unless Player 1 distributes the money 50-50, she will reject the offer.

But as in the Prisoner's Dilemma, this fails when you have no reason to expect your opponent to follow through on her precommitment. Imagine you're Player 2, playing a single Ultimatum Game against an opponent you never expect to meet again. You dutifully promise Player 1 that you will reject any offer less than 50-50. Player 1 offers 80-20 anyway. You reason “Well, my ultimatum failed. If I stick to it anyway, I walk away with nothing. I might as well admit it was a good try, give in, and take the $20. After all, rejecting the offer won't magically bring my chance at $50 back, and there aren't any other dealings with this Player 1 guy for it to influence.”

This is seemingly a rational way to think, but if Player 1 knows you're going to think that way, she offers 99-1, same as before, no matter how sincere your ultimatum sounds.

Notice all the similarities to the Prisoner's Dilemma: playing as a "rational economic agent" gets you a bad result, it looks like you can escape that bad result by making precommitments, but since the other player can't trust your precommitments, you're right back where you started

If evolutionary solutions to the Prisoners' Dilemma look like trust or friendship or altruism, solutions to the Ultimatum Game involve different emotions entirely. The Sultan presumably does not want you to elope with his daughter. He makes an ultimatum: “Touch my daughter, and I will kill you.” You elope with her anyway, and when his guards drag you back to his palace, you argue: “Killing me isn't going to reverse what happened. Your ultimatum has failed. All you can do now by beheading me is get blood all over your beautiful palace carpet, which hurts you as well as me - the equivalent of pointlessly passing up the last dollar in an Ultimatum Game where you've just been offered a 99-1 split.”

The Sultan might counter with an argument from social institutions: “If I let you go, I will look dishonorable. I will gain a reputation as someone people can mess with without any consequences. My choice isn't between bloody carpet and clean carpet, it's between bloody carpet and people respecting my orders, or clean carpet and people continuing to defy me.”

But he's much more likely to just shout an incoherent stream of dreadful Arabic curse words. Because just as friendship is the evolutionary solution to a Prisoner's Dilemma, so anger is the evolutionary solution to an Ultimatum Game. As various gurus and psychologists have observed, anger makes us irrational. But this is the good kind of irrationality; it's the kind of irrationality that makes us pass up a 99-1 split even though the decision costs us a dollar.

And if we know that humans are the kind of life-form that tends to experience anger, then if we're playing an Ultimatum Game against a human, and that human precommits to rejecting any offer less than 50-50, we're much more likely to believe her than if we were playing against a rational utility-maximizing agent - and so much more likely to give the human a fair offer.

It is distasteful and a little bit contradictory to the spirit of rationality to believe it should lose out so badly to simple emotion, and the problem might be correctable. Here we risk crossing the poorly charted border between game theory and decision theory and reaching ideas like timeless decision theory: that one should act as if one's choices determined the output of the algorithm one instantiates (or more simply, you should assume everyone like you will make the same choice you do, and take that into account when choosing.)

More practically, however, most real-world solutions to Prisoner's Dilemmas and Ultimatum Games still hinge on one of three things: threats of reciprocation when the length of the game is unknown, social institutions and reputation systems that make defection less attractive, and emotions ranging from cooperation to anger that are hard-wired into us by evolution. In the next post, we'll look at how these play out in practice.

Backward Reasoning Over Decision Trees

60 Post author: Yvain 30 June 2012 03:17AM

Game theory is the study of how rational actors interact to pursue incentives. It starts with the same questionable premises as economics: that everyone behaves rationally, that everyone is purely self-interested1, and that desires can be exactly quantified - and uses them to investigate situations of conflict and cooperation.

Here we will begin with some fairly obvious points about decision trees, but by the end we will have the tools necessary to explain a somewhat surprising finding: that giving a US president the additional power of line-item veto may in many cases make the president less able to enact her policies. Starting at the beginning:

The basic unit of game theory is the choice. Rational agents make choices in order to maximize their utility, which is sort of like a measure of how happy they are. In a one-person game, your choices affect yourself and maybe the natural environment, but nobody else. These are pretty simple to deal with:




Here we visualize a choice as a branching tree. At each branch, we choose the option with higher utility; in this case, going to the beach. Since each outcome leads to new choices, sometimes the decision trees can be longer than this:



Here's a slightly more difficult decision, denominated in money instead of utility. If you want to make as much money as possible, then your first choice - going to college or starting a minimum wage job right Now - seems to favor the more lucrative minimum wage job. But when you take Later into account, college opens up more lucrative future choices, as measured in the gray totals on the right-hand side. This illustrates the important principle of reasoning backward over decision trees. If you reason forward, taking the best option on the first choice and so on, you end up as a low-level manager. To get the real cash, you've got to start at the end - the total on the right - and then examine what choice at each branch will take you there.

This is all about as obvious as, well, not hitting yourself on the head with a hammer, so let's move on to where it really gets interesting: two-player games.



I'm playing White, and it's my move. For simplicity I consider only two options: queen takes knight and queen takes rook. The one chess book I've read values pieces in number of pawns: a knight is worth three pawns, a rook five, a queen nine. So at first glance, it looks like my best move is to take Black's rook. As for Black, I have arbitrarily singled out pawn takes pawn as her preferred move in the current position, but if I play queen takes rook, a new option opens up for her: bishop takes queen. Let's look at the decision tree:



If I foolishly play this two player game the same way I played the one-player go-to-college game, I note that the middle branch has the highest utility for White, so I take the choice that leads there: capture the rook. And then Black plays bishop takes queen, and I am left wailing and gnashing my teeth. What did I do wrong?

I should start by assuming Black will, whenever presented with a choice, take the option with the highest Black utility. Unless Black is stupid, I can cross out any branch that requires Black to play against her own interests. So now the tree looks like this:



The two realistic options are me playing queen takes rook and ending up without a queen and -4 utility, or me playing queen takes knight and ending up with a modest gain of 2 utility.

(my apologies if I've missed some obvious strategic possibility on this particular chessboard; I'm not so good at chess but hopefully the point of the example is clear.)

This method of alternating moves in a branching tree matches both our intuitive thought processes during a chess game (“Okay, if I do this, then Black's going to do this, and then I'd do this, and then...”) and the foundation of some of the algorithms chess computers like Deep Blue use. In fact, it may seem pretty obvious, or even unnecessary. But it can be used to analyze some more complicated games with counterintuitive results.

Art of Strategy describes a debate from 1990s US politics revolving around so-called “line-item veto” power, the ability to veto only one part of a bill. For example, if Congress passed a bill declaring March to be National Game Theory Month and April to be National Branching Tree Awareness Month, the President could veto only the part about April and leave March intact (as politics currently works, the President could only veto or accept the whole bill). During the '90s, President Clinton fought pretty hard for this power, which seems reasonable as it expands his options when dealing with the hostile Republican Congress.

But Dixit and Nalebuff explain that gaining line-item veto powers might hurt a President. How? Branching trees can explain.

Imagine Clinton and the Republican Congress are fighting over a budget. We can think of this as a game of sequential moves, much like chess. On its turn, Congress proposes a budget. On Clinton's turn, he either accepts or rejects the budget. A player “wins” if the budget contains their pet projects. In this game, we start with low domestic and military budgets. Clinton really wants to raise domestic spending (utility +10), and has a minor distaste for raised military spending (utility -5). Congress really wants to raise military spending (utility +10), but has a minor distaste for raised domestic spending (utility -5). The status quo is zero utility for both parties; if neither party can come to an agreement, voters get angry at them and they both lose 2 utility. Here's the tree when Clinton lacks line-item veto:



For any particular Republican choice, Clinton will never respond in a way that does not maximize his utility, so the the Republicans reason backward and arrive at something like this:



If Republicans are perfectly rational agents, they choose the second option, high domestic and high military spending, to give them their highest plausibly obtainable utility of 5.

But what if Clinton has the line-item veto? Now his options look like this:



If the Republicans stick to their previous choice of “high domestic and high military spending”, Clinton line-item vetoes the military spending, and we end up with a situation identical to the first choice: Clinton sitting on a pile of utility, and the Republicans wailing and gnashing their teeth. The Republicans need to come up with a new strategy, and their thought processes, based on Clinton as a utility-maximizer, look like this:



Here Congress's highest utility choice is to propose low domestic spending (it doesn't matter if they give more money to the military or not as this will get line-item vetoed). Let's say they propose low domestic and low military spending, and Clinton accepts. The utilities are (0, 0), and now there is much wailing and gnashing of teeth on both sides (game theorists call this a gnash equilibrium. Maybe you've heard of it.)

But now Clinton has a utility of 0, instead of a utility of 5. Giving him extra options has cost him utility! Why should this happen, and shouldn't he be able to avoid it?

This happened because Clinton's new abilities affect not only his own choices, but those of his opponents (compare Schelling: Strategies of Conflict). He may be able to deal with this if he can make the Republicans trust him.

In summary, simple sequential games can often be explored by reasoning backwards over decision trees representing the choices of the players involved. The next post will discuss simultaneous games and the concept of a Nash equilibrium.

Footnotes:

1: Game theory requires self-interest in that all players' are driven solely by their desire to maximize their own payoff in the game currently being played without regard to the welfare of other players or any external standard of fairness. However, it can also be used to describe the behavior of altruistic agents so long as their altruistic concerns are represented in the evaluation of their payoff.

Comment author: MarkusRamikin 30 June 2012 06:47:50AM *  6 points [-]

What's the benefit of using decisions trees over a table here? It seems to me it just forces you to keep clarifying about the impression they give that the second decision happens after the first and in knowledge of the first. Which was true last time you used decision trees (the chess example) so if anyone actually new to these ideas and needing to think about them is reading this, they might find that confusing.

Comment author: Yvain 03 July 2012 03:13:35AM 1 point [-]

I chose to use trees to keep it consistent with the previous representation and to show the logical painfully clearly at each step. I figured that without actually showing how at each step you could eliminate some options as impossible, it would be harder for newbies to understand.

Comment author: [deleted] 02 July 2012 08:07:55AM *  0 points [-]

Nice post, thanks for writing it up. It was interesting and very easy to read because principles were all first motivated by problems.

That said, I still feel confused. Let me try to re-frame your post in a decision theory setting: We have to make a decision, so the infinite regress of dependencies from rational players modelling each other, which doesn't generally settle to a fixed set of actions for all players who model each other naively (as shown by the penny matching example), has to be truncated some how. To do that, the game theory people decided that they'd build a new game where instead of rational players picking actions, rational players choose among strategies which return irrational players who take constant actions. These strategies make use of "biased dice", which are facts whose values the rational players are ignorant of, at least until they've selected their strategies. Each player gets their own dice of whatever biases they want, and all the dice are independent of each other. By introducing the dice, a rational player makes themself and all the other players partially uncertain about what constant action player they will select. This common enforced ignorance (or potential ignorance, since the rational players can submit pure strategies and skip the dice all together) can have some interesting properties: there will always be (for games with whatever reasonable conditions) at least one so called Nash equilibrium, which is a set of strategies returned by the rational players for which no player would unilaterally modify their decision upon learning the decisions of the others (though this and the Schelling point thing sound like an inconsistent bag of features from people who aren't sure whether the agents' decisions do or don't depend on each other).

That's as much as I got. I don't understand the conditions for a Nash equilibrium to be optimal for everybody, i.e. something the rational agents would actually all want to do instead of something that just has nice properties. You say that "purely rational players will always end up at a Nash equilibrium", but I don't think your post quite established that and I don't think it can be completely true, because of things like the self-PD. Is that what the SI's open problems of "blackmail-free equilibrium among timeless strategists" and "fair division by continuous / multiparty agents" are about maybe? That would be cool.

Also, and probably related to not having clear conditions for an optimal equilibrium, the meta-game of submitting strategies doesn't really look like it solves the infinite regress: if we find ourselves in a Nash equilibrium and agents' decisions are independent, then great, we don't have to model anything because the particular blend of utilities and enforced common ignorance means no strategy change is expected to help, but getting everyone into an equilibrium rationally (instead of hoping everyone likes the same flavor of Schelling's Heuristic Soup) still looks hard.

If Nash equilibrium is about indifference of unilateral strategy changes for independent decisions, are there ever situations where a player can switch to a strategy with equal expected utility, putting the strategy set in disequilibrium, so that other players could change their strategies, with all players improving or keeping their expected utility constant, putting the strategy set in a different equilibrium (or into a strategy set where everyone is better off, like mutual cooperation in PD)? That is, are there negotiation paths out of equilibria whose steps make no one worse off? If so, it'd then be nice to find a general selfish scheme by which players will limit themselves to those paths. This sounds like a big fun problem.

In response to comment by [deleted] on Nash Equilibria and Schelling Points
Comment author: Yvain 03 July 2012 03:09:46AM 3 points [-]

That's as much as I got. I don't understand the conditions for a Nash equilibrium to be optimal for everybody, i.e. something the rational agents would actually all want to do instead of something that just has nice properties. You say that "purely rational players will always end up at a Nash equilibrium", but I don't think your post quite established that and I don't think it can be completely true, because of things like the self-PD. Is that what the SI's open problems of "blackmail-free equilibrium among timeless strategists" and "fair division by continuous / multiparty agents" are about maybe? That would be cool.

"Optimal for everybody" is a very un-game-theoretic outlook. Game theorists think more in terms of "optimal for me, and screw the other guy". If everyone involved is totally selfish, and they expect other players to be pretty good at figuring out their strategy, and they don't have some freaky way of correlating their decisions with those of other agents like TDT, then they'll aim for a Nash equilibrium (though if there are multiple Nash equilibria, they might not hit the same one).

This fails either when agents aren't totally selfish (if, like you, they're looking for what's optimal for everyone, which is a very different problem), or if they're using an advanced decision theory to correlate their decisions, which is harder for normal people than it is for people playing against clones of themselves or superintelligences that can output their programs. I'll discuss this more later.

Also, and probably related to not having clear conditions for an optimal equilibrium, the meta-game of submitting strategies doesn't really look like it solves the infinite regress: if we find ourselves in a Nash equilibrium and agents' decisions are independent, then great, we don't have to model anything because the particular blend of utilities and enforced common ignorance means no strategy change is expected to help, but getting everyone into an equilibrium rationally (instead of hoping everyone likes the same flavor of Schelling's Heuristic Soup) still looks hard.

Agreed.

Comment author: Maelin 02 July 2012 02:13:09AM 1 point [-]

Okay, I think I get it. You're both mind-readers, and you can't go ahead until both you and the opponent have committed to your respective plans; if one of you changes your mind about the plan the other gets the opportunity to change their mind in response. But the actual coin toss occurs as part-of-the-move, not part-of-the-plan, so while you might be sad about how the coin toss plan actually pans out, there won't be any other strategy (e.g. 'Attack West') that you'd prefer to have adopted, given that the opponent would have been able to change their strategy (to e.g. 'Defend West') in response, if you had.

...I think. Wait, why wouldn't you regret staying at work then, if you know that by changing your mind your girlfriend would have a chance to change her mind, thus getting you the better outcome..?

Comment author: Yvain 03 July 2012 02:51:40AM 1 point [-]

I explained it poorly in my comment above. The mind-reading analogy is useful, but it's just an analogy. Otherwise the solution would be "Use your amazing psionic powers to level both enemy cities without leaving your room".

If I had to extend the analogy, it might be something like this: we take a pair of strategies and run two checks on it. The first check is "If your opponent's choice was fixed, and you alone had mind-reading powers, would you change your choice, knowing your opponent's?". The second check, performed in a different reality unbeknownst to you, is "If your choice was fixed, and your opponent alone had mind-reading powers, would she change her choice, knowing yours?" If the answer to both checks is "no", then you're at Nash equilibrium. You don't get to use your mind-reading powers for two-way communication.

You can do something like what you described - if you and your girlfriend realize you're playing the game above and both share the same payoff matrix, then (go home, go home) is the obvious Schelling point because it's a just plain better option, and if you have good models of each others' minds you can get there. But both that and (stay, stay) are Nash equilibria.

Comment author: Maelin 01 July 2012 02:29:33PM *  0 points [-]

Here the answer should be obvious: it doesn't matter. Flip a coin. If you flip a coin, and your opponent flips a coin, neither of you will regret your choice. Here we see a "mixed Nash equilibrium", an equilibrium reached with the help of randomness.

Hmm, I'm still not finding this clear. If I flip a coin and it comes up heads so I attack East City, and my opponent flips a coin and it comes up to defend East City, so I get zero utility and my opponent gets 1, wouldn't I regret not choosing to just attack West City instead? Or not choosing to allocate 'heads' to West City instead of East?

Is there a subtlety by what we mean by 'regret' here that I'm missing?

Comment author: Yvain 01 July 2012 02:35:49PM 3 points [-]

Here we're not thinking of your strategy as "Attack East City because the coin told me." We're thinking of your strategy as "flip a coin". The same is true of your opponent: his strategy is not "Defend East City" but "flip a coin to decide where to defend"

Suppose this scenario happened, and we offered you a do-over. You know what your opponent's strategy is going to be (flip a coin). You know your opponent is a mind-reader and will know what your strategy will be. Here your best strategy is still to flip a coin again and hope for better luck than last time.

Comment author: Yvain 01 July 2012 02:33:59AM *  3 points [-]

What is the advantage of talking about "this planet" versus standard anthropic SIA as you have used so many times on your blog and elsewhere?

I mean, I can see the disadvantages, those being that it's really hard to get a good definition of "this planet" that remains constant across universes, especially between universes with different numbers of planets, universes in which you don't exist, universes in which multiple yous exist, etc.

But with SIA, you can just rephrase it as "I was born on a certain planet, presumably selected randomly among planets in the multiverse that have life, and I call it 'this planet' because I was born there."

("This is a fertile planet, and we will thrive. We will rule over this planet, and we will call it...This Planet.")

Now on your diagram, "a planet has life" gives you a 33% chance of being in frames A, B, or C, and "This planet has life" under the previous equivalence with SIA means you choose a randomly selected pink planet and get 50% chance of being in frame A, 25% chance in frame B, and 25% chance in frame C, which justifies your statement that there should be a difference.

This also solves the scientist's problem just as dspeyer mentions.

Backward Reasoning Over Decision Trees

60 Post author: Yvain 30 June 2012 03:17AM

Game theory is the study of how rational actors interact to pursue incentives. It starts with the same questionable premises as economics: that everyone behaves rationally, that everyone is purely self-interested1, and that desires can be exactly quantified - and uses them to investigate situations of conflict and cooperation.

Here we will begin with some fairly obvious points about decision trees, but by the end we will have the tools necessary to explain a somewhat surprising finding: that giving a US president the additional power of line-item veto may in many cases make the president less able to enact her policies. Starting at the beginning:

The basic unit of game theory is the choice. Rational agents make choices in order to maximize their utility, which is sort of like a measure of how happy they are. In a one-person game, your choices affect yourself and maybe the natural environment, but nobody else. These are pretty simple to deal with:




Here we visualize a choice as a branching tree. At each branch, we choose the option with higher utility; in this case, going to the beach. Since each outcome leads to new choices, sometimes the decision trees can be longer than this:



Here's a slightly more difficult decision, denominated in money instead of utility. If you want to make as much money as possible, then your first choice - going to college or starting a minimum wage job right Now - seems to favor the more lucrative minimum wage job. But when you take Later into account, college opens up more lucrative future choices, as measured in the gray totals on the right-hand side. This illustrates the important principle of reasoning backward over decision trees. If you reason forward, taking the best option on the first choice and so on, you end up as a low-level manager. To get the real cash, you've got to start at the end - the total on the right - and then examine what choice at each branch will take you there.

This is all about as obvious as, well, not hitting yourself on the head with a hammer, so let's move on to where it really gets interesting: two-player games.



I'm playing White, and it's my move. For simplicity I consider only two options: queen takes knight and queen takes rook. The one chess book I've read values pieces in number of pawns: a knight is worth three pawns, a rook five, a queen nine. So at first glance, it looks like my best move is to take Black's rook. As for Black, I have arbitrarily singled out pawn takes pawn as her preferred move in the current position, but if I play queen takes rook, a new option opens up for her: bishop takes queen. Let's look at the decision tree:



If I foolishly play this two player game the same way I played the one-player go-to-college game, I note that the middle branch has the highest utility for White, so I take the choice that leads there: capture the rook. And then Black plays bishop takes queen, and I am left wailing and gnashing my teeth. What did I do wrong?

I should start by assuming Black will, whenever presented with a choice, take the option with the highest Black utility. Unless Black is stupid, I can cross out any branch that requires Black to play against her own interests. So now the tree looks like this:



The two realistic options are me playing queen takes rook and ending up without a queen and -4 utility, or me playing queen takes knight and ending up with a modest gain of 2 utility.

(my apologies if I've missed some obvious strategic possibility on this particular chessboard; I'm not so good at chess but hopefully the point of the example is clear.)

This method of alternating moves in a branching tree matches both our intuitive thought processes during a chess game (“Okay, if I do this, then Black's going to do this, and then I'd do this, and then...”) and the foundation of some of the algorithms chess computers like Deep Blue use. In fact, it may seem pretty obvious, or even unnecessary. But it can be used to analyze some more complicated games with counterintuitive results.

Art of Strategy describes a debate from 1990s US politics revolving around so-called “line-item veto” power, the ability to veto only one part of a bill. For example, if Congress passed a bill declaring March to be National Game Theory Month and April to be National Branching Tree Awareness Month, the President could veto only the part about April and leave March intact (as politics currently works, the President could only veto or accept the whole bill). During the '90s, President Clinton fought pretty hard for this power, which seems reasonable as it expands his options when dealing with the hostile Republican Congress.

But Dixit and Nalebuff explain that gaining line-item veto powers might hurt a President. How? Branching trees can explain.

Imagine Clinton and the Republican Congress are fighting over a budget. We can think of this as a game of sequential moves, much like chess. On its turn, Congress proposes a budget. On Clinton's turn, he either accepts or rejects the budget. A player “wins” if the budget contains their pet projects. In this game, we start with low domestic and military budgets. Clinton really wants to raise domestic spending (utility +10), and has a minor distaste for raised military spending (utility -5). Congress really wants to raise military spending (utility +10), but has a minor distaste for raised domestic spending (utility -5). The status quo is zero utility for both parties; if neither party can come to an agreement, voters get angry at them and they both lose 2 utility. Here's the tree when Clinton lacks line-item veto:



For any particular Republican choice, Clinton will never respond in a way that does not maximize his utility, so the the Republicans reason backward and arrive at something like this:



If Republicans are perfectly rational agents, they choose the second option, high domestic and high military spending, to give them their highest plausibly obtainable utility of 5.

But what if Clinton has the line-item veto? Now his options look like this:



If the Republicans stick to their previous choice of “high domestic and high military spending”, Clinton line-item vetoes the military spending, and we end up with a situation identical to the first choice: Clinton sitting on a pile of utility, and the Republicans wailing and gnashing their teeth. The Republicans need to come up with a new strategy, and their thought processes, based on Clinton as a utility-maximizer, look like this:



Here Congress's highest utility choice is to propose low domestic spending (it doesn't matter if they give more money to the military or not as this will get line-item vetoed). Let's say they propose low domestic and low military spending, and Clinton accepts. The utilities are (0, 0), and now there is much wailing and gnashing of teeth on both sides (game theorists call this a gnash equilibrium. Maybe you've heard of it.)

But now Clinton has a utility of 0, instead of a utility of 5. Giving him extra options has cost him utility! Why should this happen, and shouldn't he be able to avoid it?

This happened because Clinton's new abilities affect not only his own choices, but those of his opponents (compare Schelling: Strategies of Conflict). He may be able to deal with this if he can make the Republicans trust him.

In summary, simple sequential games can often be explored by reasoning backwards over decision trees representing the choices of the players involved. The next post will discuss simultaneous games and the concept of a Nash equilibrium.

Footnotes:

1: Game theory requires self-interest in that all players' are driven solely by their desire to maximize their own payoff in the game currently being played without regard to the welfare of other players or any external standard of fairness. However, it can also be used to describe the behavior of altruistic agents so long as their altruistic concerns are represented in the evaluation of their payoff.

Introduction to Prisoners' Dilemma

19 Post author: Yvain 30 June 2012 12:54AM

Related to: Previous posts on the Prisoners' Dilemma

Sometimes Nash equilibria just don't match our intuitive criteria for a good outcome. The classic example is the Prisoners' Dilemma.

The police arrest two criminals, Alice and Bob, on suspicion of murder. The police admit they don't have enough evidence to convict the pair of murder, but they do have enough evidence to convict them of a lesser offence, possession of a firearm. They place Alice and Bob in separate cells and offer them the following deal:

“If neither of you confess, we'll have to charge you with possession, which will land you one year in jail. But if you turn state's witness against your partner, we can convict your partner of murder and give her the full twenty year sentence; in exchange, we will let you go free. Unless, that is, both of you testify against each other; in that case, we'll give you both fifteen years.”

Alice's decision tree looks like this (note that although Alice and Bob make their decisions simultaneously, I've represented it with Alice's decision first, which is a little sketchy but should illustrate the point):



If we use the same strategy we used as a chess player, we can cross out options where Bob decides to spend extra years in jail for no personal benefit, and we're left with this:



Seen like this, the choice is clear. If you stay quiet (“cooperate”), Bob turns on you, and you are left in jail alone for twenty years, wailing and gnashing your teeth. So instead, you both turn on each other (“defect”), and end up with a sentence barely any shorter.

Another way to “prove” that defection is the “right” choice places Bob's decision first. What if you knew Bob would choose to cooperate with you? Then your choice would be between defecting and walking free, or cooperating and spending a whole year in jail - here defection wins out. But what if you knew Bob would choose to defect against you? Then your choice would be between defecting and losing fifteen years, or cooperating and losing twenty - again defection wins out. Since Bob can only either defect or cooperate, and since defection is better in both branches, “clearly” defection is the best option.

But a lot of things about this solution seem intuitively stupid. For example, when Bob goes through the same reasoning, your “rational” solution ends up with both of you in jail for fifteen years, but if you had both cooperated, you would have been out after a year. Both cooperating is better for both of you than both defecting, but you still both defect.

And if you still don't find that odd, imagine a different jurisdiction where the sentence for possession is only one day, and the police will only take a single day off your sentence for testifying against an accomplice. Now a pair of cooperators would end up with only a day in jail each, and a pair of defectors would end up with nineteen years, three hundred sixty four days each. Yet the math still tells you to defect!

Unfortunately, your cooperation only helps Bob, and Bob's cooperation only helps you. We can think of the Prisoner's Dilemma as a problem: both you and Bob prefer (cooperate, cooperate) to (defect, defect), but as it is, you're both going to end out with (defect, defect) and it doesn't seem like there's much you can do about it. To “solve” the Prisoner's Dilemma would be to come up with a way to make you and Bob pick the more desirable (cooperate, cooperate) outcome.

One proposed solution to the Prisoner's Dilemma is to iterate it - to assume it will happen multiple times in succession, as if Alice and Bob are going to commit new crimes as soon as they both get out of prison. In this case, you can threaten to reciprocate; to promise to reward cooperation with future cooperation and punish defection with future defection. Suppose Alice and Bob plan to commit two crimes, and before the first crime both promise to stay quiet on the second crime if and only if their partner stays quiet on the first. Now your decision tree as Alice looks like this:



And your calculation of Bob's thought processes go like this:



Remember that, despite how the graph looks, your first choice and Bob's first choice are simultaneous: they can't causally affect each other. So as Alice, you reason like this: On the top, Bob knows that if you testify against him, his choice will be either to testify against you (leading to the branch where you both testify against each other again next time) or to stay quiet (leading to a branch where next time he testifies against you but you stay quiet). So Bob reasons that if you testify against him, he should stay quiet this time.

On the bottom, Bob knows that if you don't testify against him, he can either testify against you (leading to the branch where you testify against him next time but he stays quiet) or stay quiet (leading to the branch where you both stay quiet again next time). Therefore, if you don't testify against him, Bob won't testify against you.

So you know that no matter what you do this time, Bob won't testify against you. That means your choice is between branches 2 and 4: Bob testifying against you next time or Bob not testifying against you next time. You prefer Branch 4, so you decide not to testify against Bob. The dilemma ends with neither of you testifying against each other in either crime, and both of you getting away with very light two year sentences.

The teeny tiny little flaw in this plan is that Bob may be a dirty rotten liar. Maybe he says he'll reciprocate, and so you both stay quiet after the first crime. Upon getting out of jail you continue your crime spree, predictably get re-arrested, and you stay quiet like you said you would to reward Bob's cooperation last time. But at the trial, you get a nasty surprise: Bob defects against you and walks free, and you end up with a twenty year sentence.

If we ratchet up to sprees of one hundred crimes and subsequent sentences (presumably committed by immortal criminals who stubbornly refuse to be cowed by the police's 100% conviction rate)  on first glance it looks like we can successfully ensure Bob's cooperation on 99 of those crimes. After all, Bob won't want to defect on crime 50, because I could punish him on crime 51. He won't want to defect on crime 99, because I could punish him on crime 100. But he will want to defect on crime 100, because he gains either way and there's nothing I can do to punish him.

Here's where it gets weird. I assume Bob is a rational utility-maximizer and so will always defect on crime 100, since it benefits him and I can't punish him for it. So since I'm also rational, I might as well also defect on crime 100; my previous incentive to cooperate was to ensure Bob's good behavior, but since Bob won't show good behavior on crime 100 no matter what I do, I might as well look after my own interests.

But if we both know that we're both going to defect on crime 100 no matter what, then there's no incentive to cooperate on crime 99. After all, the only incentive to cooperate on crime 99 was to ensure my rival's cooperation on crime 100, and since that's out of the picture anyway, I might as well shorten my sentence a little.

Sadly, this generalizes by a sort of proof by induction. If crime N will always be (defect, defect), then crime N-1 should also always be (defect, defect), which means we should defect on all of the hundred crimes in our spree.

This feat of reasoning has limited value in real life, where perfectly rational immortal criminals rarely plot in smoke-filled rooms to commit exactly one hundred crimes together; criminals who are uncertain exactly when their crime sprees will come to a close still have incentive to cooperate. But it still looks like we're going to need a better solution than simply iterating the dilemma. The next post will discuss possibilities for such a solution.

Introduction to Prisoners' Dilemma

19 Post author: Yvain 30 June 2012 12:54AM

Related to: Previous posts on the Prisoners' Dilemma

Sometimes Nash equilibria just don't match our intuitive criteria for a good outcome. The classic example is the Prisoners' Dilemma.

The police arrest two criminals, Alice and Bob, on suspicion of murder. The police admit they don't have enough evidence to convict the pair of murder, but they do have enough evidence to convict them of a lesser offence, possession of a firearm. They place Alice and Bob in separate cells and offer them the following deal:

“If neither of you confess, we'll have to charge you with possession, which will land you one year in jail. But if you turn state's witness against your partner, we can convict your partner of murder and give her the full twenty year sentence; in exchange, we will let you go free. Unless, that is, both of you testify against each other; in that case, we'll give you both fifteen years.”

Alice's decision tree looks like this (note that although Alice and Bob make their decisions simultaneously, I've represented it with Alice's decision first, which is a little sketchy but should illustrate the point):



If we use the same strategy we used as a chess player, we can cross out options where Bob decides to spend extra years in jail for no personal benefit, and we're left with this:



Seen like this, the choice is clear. If you stay quiet (“cooperate”), Bob turns on you, and you are left in jail alone for twenty years, wailing and gnashing your teeth. So instead, you both turn on each other (“defect”), and end up with a sentence barely any shorter.

Another way to “prove” that defection is the “right” choice places Bob's decision first. What if you knew Bob would choose to cooperate with you? Then your choice would be between defecting and walking free, or cooperating and spending a whole year in jail - here defection wins out. But what if you knew Bob would choose to defect against you? Then your choice would be between defecting and losing fifteen years, or cooperating and losing twenty - again defection wins out. Since Bob can only either defect or cooperate, and since defection is better in both branches, “clearly” defection is the best option.

But a lot of things about this solution seem intuitively stupid. For example, when Bob goes through the same reasoning, your “rational” solution ends up with both of you in jail for fifteen years, but if you had both cooperated, you would have been out after a year. Both cooperating is better for both of you than both defecting, but you still both defect.

And if you still don't find that odd, imagine a different jurisdiction where the sentence for possession is only one day, and the police will only take a single day off your sentence for testifying against an accomplice. Now a pair of cooperators would end up with only a day in jail each, and a pair of defectors would end up with nineteen years, three hundred sixty four days each. Yet the math still tells you to defect!

Unfortunately, your cooperation only helps Bob, and Bob's cooperation only helps you. We can think of the Prisoner's Dilemma as a problem: both you and Bob prefer (cooperate, cooperate) to (defect, defect), but as it is, you're both going to end out with (defect, defect) and it doesn't seem like there's much you can do about it. To “solve” the Prisoner's Dilemma would be to come up with a way to make you and Bob pick the more desirable (cooperate, cooperate) outcome.

One proposed solution to the Prisoner's Dilemma is to iterate it - to assume it will happen multiple times in succession, as if Alice and Bob are going to commit new crimes as soon as they both get out of prison. In this case, you can threaten to reciprocate; to promise to reward cooperation with future cooperation and punish defection with future defection. Suppose Alice and Bob plan to commit two crimes, and before the first crime both promise to stay quiet on the second crime if and only if their partner stays quiet on the first. Now your decision tree as Alice looks like this:



And your calculation of Bob's thought processes go like this:



Remember that, despite how the graph looks, your first choice and Bob's first choice are simultaneous: they can't causally affect each other. So as Alice, you reason like this: On the top, Bob knows that if you testify against him, his choice will be either to testify against you (leading to the branch where you both testify against each other again next time) or to stay quiet (leading to a branch where next time he testifies against you but you stay quiet). So Bob reasons that if you testify against him, he should stay quiet this time.

On the bottom, Bob knows that if you don't testify against him, he can either testify against you (leading to the branch where you testify against him next time but he stays quiet) or stay quiet (leading to the branch where you both stay quiet again next time). Therefore, if you don't testify against him, Bob won't testify against you.

So you know that no matter what you do this time, Bob won't testify against you. That means your choice is between branches 2 and 4: Bob testifying against you next time or Bob not testifying against you next time. You prefer Branch 4, so you decide not to testify against Bob. The dilemma ends with neither of you testifying against each other in either crime, and both of you getting away with very light two year sentences.

The teeny tiny little flaw in this plan is that Bob may be a dirty rotten liar. Maybe he says he'll reciprocate, and so you both stay quiet after the first crime. Upon getting out of jail you continue your crime spree, predictably get re-arrested, and you stay quiet like you said you would to reward Bob's cooperation last time. But at the trial, you get a nasty surprise: Bob defects against you and walks free, and you end up with a twenty year sentence.

If we ratchet up to sprees of one hundred crimes and subsequent sentences (presumably committed by immortal criminals who stubbornly refuse to be cowed by the police's 100% conviction rate)  on first glance it looks like we can successfully ensure Bob's cooperation on 99 of those crimes. After all, Bob won't want to defect on crime 50, because I could punish him on crime 51. He won't want to defect on crime 99, because I could punish him on crime 100. But he will want to defect on crime 100, because he gains either way and there's nothing I can do to punish him.

Here's where it gets weird. I assume Bob is a rational utility-maximizer and so will always defect on crime 100, since it benefits him and I can't punish him for it. So since I'm also rational, I might as well also defect on crime 100; my previous incentive to cooperate was to ensure Bob's good behavior, but since Bob won't show good behavior on crime 100 no matter what I do, I might as well look after my own interests.

But if we both know that we're both going to defect on crime 100 no matter what, then there's no incentive to cooperate on crime 99. After all, the only incentive to cooperate on crime 99 was to ensure my rival's cooperation on crime 100, and since that's out of the picture anyway, I might as well shorten my sentence a little.

Sadly, this generalizes by a sort of proof by induction. If crime N will always be (defect, defect), then crime N-1 should also always be (defect, defect), which means we should defect on all of the hundred crimes in our spree.

This feat of reasoning has limited value in real life, where perfectly rational immortal criminals rarely plot in smoke-filled rooms to commit exactly one hundred crimes together; criminals who are uncertain exactly when their crime sprees will come to a close still have incentive to cooperate. But it still looks like we're going to need a better solution than simply iterating the dilemma. The next post will discuss possibilities for such a solution.

Comment author: Maelin 29 June 2012 08:13:56AM 8 points [-]

This is a minor quibble, but while reading I got stuck at this point:

And since John Nash (remember that movie A Beautiful Mind?) proved that every game has at least one,

followed by a description of a game that didn't seem to have a Nash equilibrium and confirming text "Here there is no pure Nash equilibrium." and "So every option has someone regretting their choice, and there is no simple Nash equilibrium. What do you do?"

I kept re-reading this section, trying to work out how to reconcile these statements since it seemed like you have just offered an irrefutable counterexample to John Nash's theorem. It could use a bit of clarification (maybe something like "This game does have a Nash equilibrium, but one that is a little more subtle" or something similar.

Other than that I'm finding this sequence excellent so far.

Comment author: Yvain 30 June 2012 12:44:42AM 1 point [-]

Good point. I've clarified pure vs. mixed equilibria above.

Nash Equilibria and Schelling Points

41 Yvain 29 June 2012 02:06AM

A Nash equilibrium is an outcome in which neither player is willing to unilaterally change her strategy, and they are often applied to games in which both players move simultaneously and where decision trees are less useful.

Suppose my girlfriend and I have both lost our cell phones and cannot contact each other. Both of us would really like to spend more time at home with each other (utility 3). But both of us also have a slight preference in favor of working late and earning some overtime (utility 2). If I go home and my girlfriend's there and I can spend time with her, great. If I stay at work and make some money, that would be pretty okay too. But if I go home and my girlfriend's not there and I have to sit around alone all night, that would be the worst possible outcome (utility 1). Meanwhile, my girlfriend has the same set of preferences: she wants to spend time with me, she'd be okay with working late, but she doesn't want to sit at home alone.



This “game” has two Nash equilibria. If we both go home, neither of us regrets it: we can spend time with each other and we've both got our highest utility. If we both stay at work, again, neither of us regrets it: since my girlfriend is at work, I am glad I stayed at work instead of going home, and since I am at work, my girlfriend is glad she stayed at work instead of going home. Although we both may wish that we had both gone home, neither of us specifically regrets our own choice, given our knowledge of how the other acted.

When all players in a game are reasonable, the (apparently) rational choice will be to go for a Nash equilibrium (why would you want to make a choice you'll regret when you know what the other player chose?) And since John Nash (remember that movie A Beautiful Mind?) proved that every game has at least one, all games between well-informed rationalists (who are not also being superrational in a sense to be discussed later) should end in one of these.

What if the game seems specifically designed to thwart Nash equilibria? Suppose you are a general invading an enemy country's heartland. You can attack one of two targets, East City or West City (you declared war on them because you were offended by their uncreative toponyms). The enemy general only has enough troops to defend one of the two cities. If you attack an undefended city, you can capture it easily, but if you attack the city with the enemy army, they will successfully fight you off.



Here there is no Nash equilibrium without introducing randomness. If both you and your enemy choose to go to East City, you will regret your choice - you should have gone to West and taken it undefended. If you go to East and he goes to West, he will regret his choice - he should have gone East and stopped you in your tracks. Reverse the names, and the same is true of the branches where you go to West City. So every option has someone regretting their choice, and there is no simple Nash equilibrium. What do you do?

Here the answer should be obvious: it doesn't matter. Flip a coin. If you flip a coin, and your opponent flips a coin, neither of you will regret your choice. Here we see a "mixed Nash equilibrium", an equilibrium reached with the help of randomness.

We can formalize this further. Suppose you are attacking a different country with two new potential targets: Metropolis and Podunk. Metropolis is a rich and strategically important city (utility: 10); Podunk is an out of the way hamlet barely worth the trouble of capturing it (utility: 1).



A so-called first-level player thinks: “Well, Metropolis is a better prize, so I might as well attack that one. That way, if I win I get 10 utility instead of 1”

A second-level player thinks: “Obviously Metropolis is a better prize, so my enemy expects me to attack that one. So if I attack Podunk, he'll never see it coming and I can take the city undefended.”

A third-level player thinks: “Obviously Metropolis is a better prize, so anyone clever would never do something as obvious as attack there. They'd attack Podunk instead. But my opponent knows that, so, seeking to stay one step ahead of me, he has defended Podunk. He will never expect me to attack Metropolis, because that would be too obvious. Therefore, the city will actually be undefended, so I should take Metropolis.”

And so on ad infinitum, until you become hopelessly confused and have no choice but to spend years developing a resistance to iocane powder.

But surprisingly, there is a single best solution to this problem, even if you are playing against an opponent who, like Professor Quirrell, plays “one level higher than you.”

When the two cities were equally valuable, we solved our problem by flipping a coin. That won't be the best choice this time. Suppose we flipped a coin and attacked Metropolis when we got heads, and Podunk when we got tails. Since my opponent can predict my strategy, he would defend Metropolis every time; I am equally likely to attack Podunk and Metropolis, but taking Metropolis would cost them much more utility. My total expected utility from flipping the coin is 0.5: half the time I successfully take Podunk and gain 1 utility, and half the time I am defeated at Metropolis and gain 0.And this is not a Nash equilibrium: if I had known my opponent's strategy was to defend Metropolis every time, I would have skipped the coin flip and gone straight for Podunk.

So how can I find a Nash equilibrium? In a Nash equilibrium, I don't regret my strategy when I learn my opponent's action. If I can come up with a strategy that pays exactly the same utility whether my opponent defends Podunk or Metropolis, it will have this useful property. We'll start by supposing I am flipping a biased coin that lands on Metropolis x percent of the time, and therefore on Podunk (1-x) percent of the time. To be truly indifferent which city my opponent defends, 10x (the utility my strategy earns when my opponent leaves Metropolis undefended) should equal 1(1-x) (the utility my strategy earns when my opponent leaves Podunk undefended). Some quick algebra finds that 10x = 1(1-x) is satisfied by x = 1/11. So I should attack Metropolis 1/11 of the time and Podunk 10/11 of the time.

My opponent, going through a similar process, comes up with the suspiciously similar result that he should defend Metropolis 10/11 of the time, and Podunk 1/11 of the time.

If we both pursue our chosen strategies, I gain an average 0.9090... utility each round, soundly beating my previous record of 0.5, and my opponent suspiciously loses an average -.9090 utility. It turns out there is no other strategy I can use to consistently do better than this when my opponent is playing optimally, and that even if I knew my opponent's strategy I would not be able to come up with a better strategy to beat it. It also turns out that there is no other strategy my opponent can use to consistently do better than this if I am playing optimally, and that my opponent, upon learning my strategy, doesn't regret his strategy either.

In The Art of Strategy, Dixit and Nalebuff cite a real-life application of the same principle in, of all things, penalty kicks in soccer. A right-footed kicker has a better chance of success if he kicks to the right, but a smart goalie can predict that and will defend to the right; a player expecting this can accept a less spectacular kick to the left if he thinks the left will be undefended, but a very smart goalie can predict this too, and so on. Economist Ignacio Palacios-Huerta laboriously analyzed the success rates of various kickers and goalies on the field, and found that they actually pursued a mixed strategy generally within 2% of the game theoretic ideal, proving that people are pretty good at doing these kinds of calculations unconsciously.

So every game really does have at least one Nash equilibrium, even if it's only a mixed strategy. But some games can have many, many more. Recall the situation between me and my girlfriend:



There are two Nash equilibria: both of us working late, and both of us going home. If there were only one equilibrium, and we were both confident in each other's rationality, we could choose that one and there would be no further problem. But in fact this game does present a problem: intuitively it seems like we might still make a mistake and end up in different places.

Here we might be tempted to just leave it to chance; after all, there's a 50% probability we'll both end up choosing the same activity. But other games might have thousands or millions of possible equilibria and so will require a more refined approach.

Art of Strategy describes a game show in which two strangers were separately taken to random places in New York and promised a prize if they could successfully meet up; they had no communication with one another and no clues about how such a meeting was to take place. Here there are a nearly infinite number of possible choices: they could both meet at the corner of First Street and First Avenue at 1 PM, they could both meet at First Street and Second Avenue at 1:05 PM, etc. Since neither party would regret their actions (if I went to First and First at 1 and found you there, I would be thrilled) these are all Nash equilibria.

Despite this mind-boggling array of possibilities, in fact all six episodes of this particular game ended with the two contestants meeting successfully after only a few days. The most popular meeting site was the Empire State Building at noon.

How did they do it? The world-famous Empire State Building is what game theorists call focal: it stands out as a natural and obvious target for coordination. Likewise noon, classically considered the very middle of the day, is a focal point in time. These focal points, also called Schelling points after theorist Thomas Schelling who discovered them, provide an obvious target for coordination attempts.

What makes a Schelling point? The most important factor is that it be special. The Empire State Building, depending on when the show took place, may have been the tallest building in New York; noon is the only time that fits the criteria of “exactly in the middle of the day”, except maybe midnight when people would be expected to be too sleepy to meet up properly.

Of course, specialness, like beauty, is in the eye of the beholder. David Friedman writes:

Two people are separately confronted with the list of numbers [2, 5, 9, 25, 69, 73, 82, 96, 100, 126, 150 ] and offered a reward if they independently choose the same number. If the two are mathematicians, it is likely that they will both choose 2—the only even prime. Non-mathematicians are likely to choose 100—a number which seems, to the mathematicians, no more unique than the other two exact squares. Illiterates might agree on 69, because of its peculiar symmetry—as would, for a different reason, those whose interest in numbers is more prurient than mathematical.

A recent open thread comment pointed out that you can justify anything with “for decision-theoretic reasons” or “due to meta-level concerns”. I humbly propose adding “as a Schelling point” to this list, except that the list is tongue-in-cheek and Schelling points really do explain almost everything - stock markets, national borders, marriagesprivate property, religions, fashion, political parties, peace treaties, social networks, software platforms and languages all involve or are based upon Schelling points. In fact, whenever something has “symbolic value” a Schelling point is likely to be involved in some way. I hope to expand on this point a bit more later.

Sequential games can include one more method of choosing between Nash equilibria: the idea of a subgame-perfect equilibrium, a special kind of Nash equlibrium that remains a Nash equilibrium for every subgame of the original game. In more intuitive terms, this equilibrium means that even in a long multiple-move game no one at any point makes a decision that goes against their best interests (remember the example from the last post, where we crossed out the branches in which Clinton made implausible choices that failed to maximize his utility?) Some games have multiple Nash equilibria but only one subgame-perfect one; we'll examine this idea further when we get to the iterated prisoners' dilemma and ultimatum game.

In conclusion, every game has at least one Nash equilibrium, a point at which neither player regrets her strategy even when she knows the other player's strategy. Some equilibria are simple choices, others involve plans to make choices randomly according to certain criteria. Purely rational players will always end up at a Nash equilibrium, but many games will have multiple possible equilibria. If players are trying to coordinate, they may land at a Schelling point, an equilibria which stands out as special in some way.

Comment author: Lukas_Gloor 28 June 2012 03:03:25PM 5 points [-]

If we don't allow potential people to carry weight, and if we are preference rather than hedonic utilitarians, then the only thing we are checking when deciding to create all these new people is whether or not existing people would prefer to do so.

That's Peter Singer's view, prior-existence instead of total. A problem here seems to be that creating a being in intense suffering would be ethically neutral, and if even the slightest preference for doing so exists, and if there were no resource trade-offs in regard to other preferences, then creating that miserable being would be the right thing to do. One can argue that in the first millisecond after creating the miserable being, one would be obliged to kill it, and that, foreseeing this, one ought not have created it in the first place. But that seems not very elegant. And one could further imagine creating the being somewhere unreachable, where it's impossible to kill it afterwards.

One can avoid this conclusion by axiomatically stating that it is bad to bring into existence a being with a "life not worth living". But that still leaves problems, for one thing, it seems ad hoc, and for another, it would then not matter whether one brings a happy child into existence or one with a neutral life, and that again seems highly counterintuitive.

The only way to solve this, as I see it, is to count all unsatisfied preferences negatively. You'd end up with negative total preference-utiltiarianism, which usually has quite strong reasons against bringing beings into existence. Depending on how much pre-existing beings want to have children, it wouldn't necessarily entail complete anti-natalism, but the overall goal would at some point be a universe without unsatisfied preferences. Or is there another way out?

Comment author: Yvain 29 June 2012 12:19:07AM 2 points [-]

Thank you. Apparently total utilitarianism really is scary, and I had routed around it by replacing it with something more useable and assuming that was what everyone else meant when they said "total utilitarianism".

Comment author: beoShaffer 28 June 2012 04:45:14AM 9 points [-]

Pretty good so far. I personally felt like the line-item example was pretty illuminative, but using real political examples instead of sticking with the national game theory month approach is a bit iffy. In particular the line

If Republicans are perfectly rational agents (a sentiment, I admit, open to doubt)

Can easily be interpreted as an attack on republicans.

Comment author: Yvain 28 June 2012 11:54:55PM 6 points [-]

Removed.

Comment author: kilobug 28 June 2012 08:04:22AM 15 points [-]

Good introduction, nothing new (but the "line-item veto" example is a nice one), but still good to have around.

One thing that made me "wail and gnash" a little :

that everyone is purely self-interested

That's not really the case. Game theory usually consider that everyone is an utility maximizer, but nothing says that the utility function has to be selfish. Utility function can factor well-being and happiness of others in it.

You can apply game theory in cases like a parent-child relation, in which the parent and the child disagree, but the parent still is motivated with the interest of the child. Even in more classical cases, nothing forces the utility function to be selfish and not value the other well-being. Game theory only apply when the agents have different goals, but it can just be "I value my own well-being twice as much as the well-being of the other", which is not "purely self-interested".

It makes me "wail and gnash" because it's a very frequent cliché that rationalists and utility maximizer are necessarily selfish and don't care about others, and it's a cliché we should fight. That say, I understand the a whole part of game theory is about showing how even pure selfishness can, in some cases, lead to cooperation being the best solution. Your "line-item veto" is a good example of it, Clinton and Congress can still cooperate to get the previous 5-5 equilibrium, if they trust each others, and this is an IPD at the end.

Comment author: Yvain 28 June 2012 11:54:16PM 1 point [-]

See my response to steven0461 and my footnote. Yes, we will eventually be able to derive cooperation, but we will derive it by starting with selfish assumptions.

Comment author: steven0461 28 June 2012 05:44:45AM 9 points [-]

It starts with the same questionable premises as economics: that everyone behaves rationally, that everyone is purely self-interested, and that desires can be easily and exactly quantified

I agree that it assumes that everyone behaves rationally and that desires can be exactly quantified, but I don't see how it assumes that everyone is purely self-interested or that desires can be easily quantified.

Comment author: Yvain 28 June 2012 11:52:58PM 0 points [-]

By "easily quantified", I meant that there is exactly one possible quantification system that is known by and agreed to by both players, as opposed to the mess going on in real life as is being discussed on Stuart_Armstrong's thread. But I've deleted that part as unclear.

By "self-interested", I mean for example that the Nash equilibrium for Prisoners' Dilemma is (D,D) because we assume both are trying to minimize their sentence, and we don't have to worry about one of them trying to lessen the other's sentence at his own expense because the other prisoner has a wife and kids, or about them wanting to lengthen their own sentence to pay their debt to society. I agree that one could use the same math to analyze someone who is only playing a game to donate the payoff to charity, or something. I've added a footnote clarifying this further.

Comment author: Yvain 28 June 2012 04:19:09AM *  4 points [-]

Upvoted, but as someone who, without quite being a total utilitarian, at least hopes someone might be able to rescue total utilitarianism, I don't find much to disagree with here. Points 1, 4, 5, and 6 are arguments against certain claims that total utilitarianism should be obviously true, but not arguments that it doesn't happen to be true.

Point 2 states that total utilitarianism won't magically implement itself and requires "technology" rather than philosophy; that is, people have to come up with specific contingent techniques of estimating utility, rather than just reading it off via a simple method which can be proven mathematically perfect. But we have some Stone Age utility-comparing technologies like money and the popular vote, and QALYs might be metaphorically a Bronze Age technology. I suppose I take it on faith that there's a lot of room for more advanced technology before we hit mathematical limits.

That leaves the introductory paragraph and Point 3 as the only places I still disagree:

In total utilitarianism, it is a morally neutral act to kill someone (in a painless and unexpected manner) and creating/giving birth to another being of comparable happiness (or preference satisfaction or welfare).

In hedonic utilitarianism, yes. Are you making this claim for preference utilitarianism as well? If so, on what basis? If we don't give credit for creating potential people, isn't most people's preference not to be killed enough to stop preference utilitarians from killing them?

And you also have to be certain that your theory does not allow path dependency. One can take the perfectly valid position that "If there were an existing poorer population, then the right thing to do would be to redistribute wealth, and thus lose the last copy of Akira. However, currently there is no existing poor population, hence I would oppose it coming into being, precisely because it would result in the lose of Akira." You can reject this type of reasoning, and a variety of others that block the repugnant conclusion at some stage of the chain (the Stanford Encyclopaedia of Philosophy has a good entry on the Repugnant Conclusion and the arguments surrounding it). But most reasons for doing so already pre-suppose total utilitarianism. In that case, you cannot use the above as an argument for your theory.

Can you explain this further? If we don't allow potential people to carry weight, and if we are preference rather than hedonic utilitarians, then the only thing we are checking when deciding to create all these new people is whether or not existing people would prefer to do so.

The fact that the repugnant conclusion has "repugnant" right in the name suggests that most people don't want it. Therefore if total utilitarianism is about satisfying the preferences of as many people as possible much as possible, and it results in a conclusion nobody prefers, that should be a red flag.

If existing people understand the repugnant conclusion, then they will understand it is a likely consequence of creating all these people is that the world loses most of its culture and happiness, and when we aggregate their preferences they will vote against doing so.

So I don't see what you mean when you say this reasoning "pre-supposes total utiltarianism". It presupposes people's intuitive moral preferences for a happy world full of culture to a just-barely-not-unhappy-world without, and it pretends we can solve the aggregation problem, but where's the vicious self-reference?

Introduction to Game Theory: Sequence Guide

42 Yvain 28 June 2012 03:32AM

This sequence of posts is a primer on game theory intended at an introductory level. Because it is introductory, Less Wrong veterans may find some parts boring, obvious, or simplistic - although hopefully nothing is so simplistic as to be outright wrong.

Parts of this sequence draw heavily upon material from The Art of Strategy by Avinash Dixit and Barry Nalebuff, and it may in part be considered a (very favorable) review of the book accompanied by an exploration of its content. I have tried to include enough material to be useful, but not so much material that it becomes a plagiarism rather than a review (it's probably a bad idea to pick a legal fight with people who write books called The Art of Strategy.) Therefore, for the most complete and engaging presentation of this material, I highly recommend the original book.

All posts will be linked from here as they go up:

1. Introduction to Game Theory: Sequence Guide
2. Backward Reasoning Over Decision Trees
3. Nash Equilibria and Schelling Points
4. Introduction to Prisoners' Dilemma
5. Real World Solutions to Prisoners' Dilemmas
6. Interlude for Behavioral Economics
7. What Is Signaling, Really?
8. Bargaining and Auctions
9. Imperfect Voting Systems
10. Game Theory As A Dark Art

Special thanks to Luke for his book recommendation and his strong encouragement to write this.

Comment author: Yvain 11 April 2012 11:09:44AM *  17 points [-]

Nicholas Flamel (born 1340) could be almost as good a source of ancient spells lost to the Interdict of Merlin as Slytherin's Monster (exact creation date unknown, but Godric Gryffindor was alive in 1202 and Slytherin was a contemporary). He also seems to be dependent on Albus Dumbledore for protection; maybe it's time Dumbledore called in some quid pro quo if he hasn't already?

Comment author: XiXiDu 05 April 2012 11:53:02AM 10 points [-]

If you don't care about measure then why try to solve friendly AI? In a VERY BIG world some AI's will turn out to be friendly.

Comment author: Yvain 07 April 2012 03:20:25PM 2 points [-]

I'm uncertain of many-worlds, Big World theory, and the reductionist view of identity as not linked to a particular body. If any of those fail, then if this body dies I'm just plain dead, and same for everyone else.

And there are many ways that an unfriendly AI - or no AI at all - could screw things up besides killing everyone instantly.

Comment author: Benquo 05 April 2012 02:45:42PM *  2 points [-]

Right now I don't go to bed at night weeping that my father only met my mother through a series of unlikely events and so most universes probably don't contain me; I'm not sure why I should do so after having been resurrected in the far future.

Because you understand that you can't change it:

For nothing is more certain, than that despair has almost the same effect upon us with enjoyment, and that we are no sooner acquainted with the impossibility of satisfying any desire, than the desire itself vanishes.

I don't think this claim is true in all cases, but it seems plausible here. There's no adaptive value to learning about "risks" you were subject to before you were able to do anything about them, so I don't feel how bad it was that I almost wasn't born as myself. But I did feel a kind of possible-worlds regret when I stepped out into the street without looking and almost got hit by a bus, since in some likely possible worlds I did get hit by that bus.

Comment author: Yvain 07 April 2012 03:09:28PM 3 points [-]

Because you understand that you can't change it

As I said below, if Omega offered me a financial deal to increase my measure - $100 per alternate Everett branch to make my father and mother meet and conceive me - this wouldn't seem like a remotely good use of my money.

Comment author: pleeppleep 05 April 2012 09:04:38PM *  10 points [-]

I have a feeling that I'm missing something and that this is going to be downvoted, but I still have to ask. In the event that a big universe exists, there are numerous people almost exactly like me going about their business. My problem is that what i would call my consciousness doesn't seem to experience their actions. This would seem to me like there is some factor in my existence that is not present in theirs. If a perfect clone was sitting next to me, I wouldn't be able to see my computer through his eyes. I would continue to see it through mine. This chain of experience is the thing I care most to preserve. I have interest in the continued existence of people like me, but for separate reasons.

I know the idea of an "inner listener" is false, but the sensation of such a thing and a continuous stream of experience do exist. I am emotionally tied to those perceptions. I don't know how enthusiastically I can look forward to the future if I won't be able to experience it any more than I can the nearest parallel universe.

Comment author: Yvain 07 April 2012 03:06:46PM 16 points [-]

This chain of experience is the thing I care most to preserve.

Okay, think of it this way.

You go to sleep tonight, your "chain of experience" is briefly broken. You wake up tomorrow morning, chain of experience is back, you're happy.

But what makes you say "chain of experience is back"? Only that a human being wakes up, notices it has the memories of being pleeppleep, and says "Hey, my chain of experience is back! Good!"

Suppose Omega killed you in your sleep, then created a perfect clone of you. The perfect clone would wake up, notice it has the memories of being pleeppleep, and say "Hey, my chain of experience is back! Good!" Then it would continue living your life.

Right now you have zero evidence that Omega hasn't actually done this to you every single night of your life. So the idea of a "chain of experience", except as another word for your memories, is pretty tenuous.

And if I told you today that Omega had really been doing this to you your whole life, then you would be really scared before going to sleep tonight, but eventually you'd have to do it. And then the next day, your clone would still be pretty scared before going to sleep, but he'd do it too. And by the thousandth day, you'd probably have forgotten all about it except when someone reminds you.

(what if it were every time you blinked, instead of every time you slept?)

Since this would be totally indistinguishable from the way we are right now, and since there's no logical basis for me-ness, at some point you just have to think, "screw it, there's no continuity of experience or identity and I don't really care", at least as regards blinking and sleeping.

Cryonicists say we can extend this indifference to freezing and thawing. I'm saying that as long as we're extending it, might as well extend it all the way.

Comment author: wedrifid 05 April 2012 03:00:46PM 8 points [-]

Quantum suicide seems like a good idea to me if we know that the assumptions behind it (both quantum and identity-related) are true, if we're purely selfish (eg don't care about the bereaved left behind), and if we don't assume our actions are sufficiently correlated with those of others to make everyone try quantum suicide and end up all alone in our own personal Everett branch.

Fortunately, if you combine the second and third potential problems you end up with a solution that eliminates both of them. Then you just have the engineering problem involved in building a bigger death box.

However, I might have the same "It's a good idea, but I am going to refuse to do this for reasons of personal sanity" reaction as I have with Pascal's Mugging.

I hope so. Your position is entirely consistent - I cannot fault it on objective grounds and what you say in your post does directly imply what you confirm in your comment. That said, the preferences you declare here are vastly different to those that I consider 'normal' and so there remains the sneaking suspicion that you are wrong about what you want. That is, that you incorrectly extrapolate your volition.

On the other hand the existence of people with the preferences you describe here is a great potential boon to the rest of us. Whenever parties have vastly different values there is the potential for trade between them. And the difference in values between those that care about measure and those that don't rounds off to absolute. When you act on your preferences we can essentially just inherit all of your stuff in exchange for a (from our perspective) token probabilistic payout. Everybody wins!

Comment author: Yvain 07 April 2012 11:00:33AM 1 point [-]

So, if Omega was willing to put a copy of you in an Everett branch that didn't already have one, how much money would you be willing to bid for this service?

If Omega was going to charge $100, and the offer remained open for as many Everett branches as you wanted, how many $100s would you give Omega?

Comment author: wedrifid 05 April 2012 02:35:40AM 23 points [-]

The only downside I can see here is that I have less measure (meaning I exist in a lower proportion of worlds) than if I had signed up for cryonics directly. This might be a problem if I think that my existence benefits others - but I don't think I should be concerned for my own sake.

Yvain is quantum suicidal? I didn't expect that! People whose preferences 'add up to normal' do (implicitly) care about measure.

Comment author: Yvain 05 April 2012 11:20:39AM 7 points [-]

Quantum suicide seems like a good idea to me if we know that the assumptions behind it (both quantum and identity-related) are true, if we're purely selfish (eg don't care about the bereaved left behind), and if we don't assume our actions are sufficiently correlated with those of others to make everyone try quantum suicide and end up all alone in our own personal Everett branch.

However, I might have the same "It's a good idea, but I am going to refuse to do this for reasons of personal sanity" reaction as I have with Pascal's Mugging.

Comment author: Mitchell_Porter 05 April 2012 03:08:13AM 11 points [-]

If all copies count as you, then that includes Boltzmann brains who die in the vacuum a second after their formation and copies of you who awaken inside a personally dedicated hell. And this is supposed to provide hope?

There is clearly a sense in which you do not experience what your copies experience. The instance of you who dies in a car crash on the way to your wedding never experiences the wedding itself; that is experienced by the second instance, created from a backup a few weeks later.

Any extension of identity beyond the "current instance" level is therefore an act of imagination or chosen affiliation. Identifying with your copies and almost-copies scattered throughout the multiverse, identifying with your descendants, and identifying with all beings who ever live, all have this in common - "you", defined in the broad sense supplied by your expansive concept of identity, will experience things that "you", defined in the narrow but practical sense of your local instance, will never experience.

Since it is a contradiction to say that you will experience things that you will never experience, it is desirable to perceive very clearly that these expansive identifications are being made by one local instance that is choosing to regard a multiplicity of other distinct beings as other parts of its extended self. Of course, once you perceive this distinction, between local self and global self, and especially once you notice that the same local self can have arbitrarily expansive or delimited beliefs about who gets to be a part of its global self... you might begin to doubt the meaningfulnss of any notion of global self other than "the whole of reality", or indeed you might doubt the meaningfulness of any notion of "global self" at all. Perhaps in reality you are just your local self and that's it; all other identifications are fantasy.

When that attitude is taken to its limit, it usually leads to disconnection between one moment and the next. Each moment's experience is only had by that momentary self. You could make a slogan out of it: "instances are instantaneous", meaning that if you apply this philosophy consistently, you have to deny that the "local self" extends in time.

But this part I don't believe, because I do believe that experiences are extended in time. There is such a thing as change, the flow of one moment into the next, and not just a static difference between static moments each containing its own encapsulated illusion of flow-connectedness to other moments. The reduction of time to simply another spatial coordinate, and the consequent relegation of the experience of time passing to the category of illusion, results from the cultural hypertrophy (that's the opposite of atrophy) of "logical perception" in scientific culture, at the expense of more "phenomenological" capacities, like a sensitivity to the actual form of consciousness. If people took appearances more seriously, their response to the difficulties of reconciling them with scientific theory would be to look for a better theory, not to call the appearances illusory or nonexistent. It's not at all easy even to get the ontology of appearance right, let alone to conceptually reconstruct the ontology by means of which we understand our mathematical theories of nature, so as to include the ontology implied by appearance.

In fact, an extra layer of difficulty has been added by the attempted reduction of epistemology to computation - it means that the epistemological claims of phenomenology, e.g. that we can know that time really passes or that change is real, struggle to get a hearing. Computational epistemology in its existing forms presupposes an inadequate ontology, and therefore offers a new, methodological barrier to any truth from outside that ontology. One needs to remember that computation is about state transitions in state machines, and says nothing about the "intrinsic nature" of those states or how that intrinsic nature may be related to the causality of the state transitions. So any ontology featuring causal interactions between things with states contains computation, in the same way that any ontology containing multiple things contains arithmetic; but you can't bootstrap your way from computation to ontology, just as you can't bootstrap your way from arithmetic to physics.

In my polemic I have strayed far from the original topic of Yvain's post, but any discussion of the ontology of persons eventually has to tackle these "hard problems".

Comment author: Yvain 05 April 2012 11:17:51AM 15 points [-]

It doesn't seem too much more distressing to believe that there are copies of me being tortured right now, than to believe that there are currently people in North Korea being tortured right now, or other similarly unpleasant facts everyone agrees to be true.

There's a distinction between intuitive identity - my ability to get really upset about the idea that me-ten-minutes-from-now will be tortured - and philosophical identity - an ability to worry slightly about the idea that a copy of me in another universe is getting tortured. This difference isn't just instrumentally based on the fact that it's easier for me to save me-ten-minutes-from-now than me-in-another-universe; even if I were offered some opportunity to help me-in-another-universe, I would feel obligated to do so only on grounds of charity, not on grounds of selfishness. I'd ground that as some mental program that intuitively makes me care about me-ten-minutes-from-now which is much stronger than whatever rational kinship I can muster with me-in-another-universe. This mental program seems pretty good at dealing with minor breaks in continuity like sleep or coma.

The problem is, once death comes into the picture, the mental program can't carry on business as usual - there won't be any "me-ten-minutes-from-now". And one reaction is to automatically switch allegiance to the closest copy of me - for example, cryonically-resurrected-me-a-century-from-now. I don't think this allegiance-switching has any fundamental ontological basis, but I'm not prepared to say it's stupid either. My point here is only that once you're okay with switching allegiances, you might as well do it to the nearest other pre-existing copy of you, rather than go through all the trouble of creating a new one.

I agree that we can't ground identity in individual moments. For one thing, the only reasonable candidate for "moment" is the Planck time, and there's no experience that can happen on that short an interval. For another, static experiences don't seem to be conscious: if I were frozen in time, I couldn't think "Darnit, I'm frozen in time now!" because that thought involves a state change. I think this is what you're saying your third to last paragraph but I'm not sure.

I'm leaning towards saying my identification with self-at-the-present-moment isn't any more interesting or fundamental than my artificially created identification with me-ten-minutes-from-now, and that a feeling of being in the present is just a basis for other computational processes. As far as I can understand, this doesn't seem to be your solution at all.

Do any of the various theories marketed as "timeless" here claim that the belief in a present moment is purely indexical - that is, a function of the randomly chosen observer-moment currently experienced as "me" being Yvain(2012) as opposed to Yvain(2013), in the same sense that seeing a quantum coin come up heads instead of tails is indexical? It seems like an elegant idea and would be relevant to this discussion.

Comment author: DanielLC 05 April 2012 12:30:59AM 0 points [-]

Why do you care if you continue to exist?

Also, if you don't care about your measure, than why is the Big Universe even necessary? You already know you have a measure of at least how long you've been alive. You can't cease to have ever existed.

Comment author: Yvain 05 April 2012 10:11:21AM 1 point [-]

The main thrust of this post is that assumptions sufficient to support cryonics are also sufficient to survive without cryonics.

The "why do you care if you continue to exist?" question is just part of the background assumption - people sign up for cryonics because they care about continuing to exist. All I'm saying is that if you believe that, you can get the same results by not signing up for cryonics, maybe.

Comment author: steven0461 04 April 2012 11:42:20PM 0 points [-]

This is a restatement of quantum immortality, right?

Comment author: Yvain 04 April 2012 11:44:51PM 0 points [-]

I'm not sure why you crossed that out; it more or less is. The only new part is that I've never heard anyone discuss the relevance to cryonics before.

Cryonics without freezers: resurrection possibilities in a Big World

40 Yvain 04 April 2012 10:48PM

And fear not lest Existence closing your
Account, should lose, or know the type no more;
The Eternal Saki from the Bowl has pour'd
Millions of Bubbles like us, and will pour.

When You and I behind the Veil are past,
Oh, but the long long while the World shall last,
Which of our Coming and Departure heeds
As much as Ocean of a pebble-cast.

    -- Omar Khayyam, Rubaiyat

 

A CONSEQUENTIALIST VIEW OF IDENTITY

The typical argument for cryonics says that if we can preserve brain data, one day we may be able to recreate a functioning brain and bring the dead back to life.

The typical argument against cryonics says that even if we could do that, the recreation wouldn't be "you". It would be someone who thinks and acts exactly like you.

The typical response to the typical argument against cryonics says that identity isn't in specific atoms, so it's probably in algorithms, and the recreation would have the same mental algorithms as you and so be you. The gap in consciousness of however many centuries is no more significant than the gap in consciousness between going to bed at night and waking up in the morning, or the gap between going into a coma and coming out of one.

We can call this a "consequentialist" view of identity, because it's a lot like the consequentialist views of morality. Whether a person is "me" isn't a function of how we got to that person, but only of where that person is right now: that is, how similar that person's thoughts and actions are to my own. It doesn't matter if we got to him by having me go to sleep and wake up as him, or got to him by having aliens disassemble my brain and then simulate it on a cellular automaton. If he thinks like me, he's me.

A corollary of the consequentialist view of identity says that if someone wants to create fifty perfect copies of me, all fifty will "be me" in whatever sense that means something.

GRADATIONS OF IDENTITY

An argument against cryonics I have never heard, but which must exist somewhere, says that even the best human technology is imperfect, and likely a few atoms here and there - or even a few entire neurons - will end up out of place. Therefore, the recreation will not be you, but someone very very similar to you.

And the response to this argument is "Who cares?" If by "me" you mean Yvain as of 10:20 PM 4th April 2012, then even Yvain as of 10:30 is going to have some serious differences at the atomic scale. Since I don't consider myself a different person every ten minutes, I shouldn't consider myself a different person if the resurrection-machine misplaces a few cells here or there.

But this is a slippery slope. If my recreation is exactly like me except for one neuron, is he the same person? Signs point to yes. What about five neurons? Five million? Or on a functional level, what if he blinked at exactly one point where I would not have done so? What if he prefers a different flavor of ice cream? What if he has exactly the same memories as I do, except for the outcome of one first-grade spelling bee I haven't thought about in years anyway? What if he is a Hindu fundamentalist?

If we're going to take a consequentialist view of identity, then my continued ability to identify with myself even if I naturally switch ice cream preferences suggests I should identify with a botched resurrection who also switches ice cream preferences. The only solution here that really makes sense is to view identity in shades of gray instead of black-and-white. An exact clone is more me than a clone with different ice cream preferences, who is more me than a clone who is a Hindu fundamentalist, who is more me than LeBron James is.

BIG WORLDS

There are various theories lumped together under the title "big world".

The simplest is the theory that the universe (or multiverse) is Very Very Big. Although the universe is probably only 15 billion years old, which means the visible universe is only 30 billion light years in size, inflation allows the entire universe to get around the speed of light restriction; it could be very large or possibly infinite. I don't have the numbers available, but I remember a back of the envelope calculation being posted on Less Wrong once about exactly how big the universe would have to be to contain repeating patches of about the size of the Earth. That is, just as the first ten digits of pi, 3141592653, must repeat somewhere else in pi because pi is infinite and patternless, and just as I would believe this with high probability even if pi were not infinite but just very very large, so the arrangement of atoms that make up Earth would recur in an infinite or very very large universe. This arrangement would obviously include you, exactly as you are now. A much larger class of Earth-sized patches would include slightly different versions of you like the one with different ice cream preferences. This would also work, as Omar Khayyam mentioned in the quote at the top, if the universe were to last forever or a very very long time.

The second type of "big world" is the one posited by the Many Worlds theory of quantum mechanics, in which each quantum event causes the Universe to split into several branches. Because quantum events determine larger-level events, and because each branch continues branching, some these branches could be similar to our universe but with observable macro-scale differences. For example, there might be a branch in which you are the President of the United States, or the Pope, or died as an infant. Although this sounds like a silly popular science version of the principle, I don't think it's unfair or incorrect.

The third type of "big world" is modal realism: the belief that all possible worlds exist, maybe in proportion to their simplicity (whatever that means). We notice the existence of our own world only for indexical reasons: that is, just as there are many countries, but when I look around me I only see my own; so there are many possibilities, but when I look around me I only see my own. If this is true, it is not only possible but certain that there is a world where I am Pope and so on.

There are other types of "big worlds" that I won't get into here, but if any type at all is correct, then there should be very many copies of me or people very much like me running around.

CRYONICS WITHOUT FREEZERS

Cryonicists say that if you freeze your brain, you may experience "waking up" a few centuries later when someone uses the brain to create a perfect copy of you.

But whether or not you freeze your brain, a Big World is creating perfect copies of you all the time. The consequentialist view of identity says that your causal connection with these copies is unnecessary for them to be you. So why should a copy of you created by a far-future cryonicist with access to your brain be better able to "resurrect" you than a copy of you that comes to exist for some other reason?

For example, suppose I choose not to sign up for cryonics, have a sudden heart attack, and die in my sleep. Somewhere in a Big World, there is someone exactly like me except that they didn't have the heart attack and they wake up healthy the next morning.

The cryonicists believe that having a healthy copy of you come into existence after you die is sufficient for you to "wake up" as that copy. So why wouldn't I "wake up" as the healthy, heart-attack-free version of me in the universe next door?

Or: suppose that a Friendly AI fills a human-sized three-dimensional grid with atoms, using a quantum dice to determine which atom occupies each "pixel" in the grid. This splits the universe into as many branches as there are possible permutations of the grid (presumably a lot) and in one of those branches, the AI's experiment creates a perfect copy of me at the moment of my death, except healthy. If creating a perfect copy of me causes my "resurrection", then that AI has just resurrected me as surely as cryonics would have.

The only downside I can see here is that I have less measure (meaning I exist in a lower proportion of worlds) than if I had signed up for cryonics directly. This might be a problem if I think that my existence benefits others - but I don't think I should be concerned for my own sake. Right now I don't go to bed at night weeping that my father only met my mother through a series of unlikely events and so most universes probably don't contain me; I'm not sure why I should do so after having been resurrected in the far future.

RESURRECTION AS SOMEONE ELSE

What if the speculative theories involved in Big Worlds all turn out to be false? All hope is still not lost.

Above I wrote:

An exact clone is more me than a clone with different ice cream preferences, who is more me than a clone who is a Hindu fundamentalist, who is more me than LeBron James is.

I used LeBron James because from what I know about him, he's quite different from me. But what if I had used someone else? One thing I learned upon discovering Less Wrong is that I had previously underestimated just how many people out there are *really similar to me*, even down to weird interests, personality quirks, and sense of humor. So let's take the person living in 2050 who is most similar to me now. I can think of several people on this site alone who would make a pretty impressive lower bound on how similar the most similar person to me would have to be.

In what way is this person waking up on the morning of January 1 2050 equivalent to me being sort of resurrected? What if this person is more similar to Yvain(2012) than Yvain(1995) is? What if I signed up for cryonics, died tomorrow, and was resurrected in 2050 by a process about as lossy as the difference between me and this person?

SUMMARY

Personal identity remains confusing. But some of the assumptions cryonicists make are, in certain situations, sufficient to guarantee personal survival after death without cryonics.

Comment author: Yvain 02 April 2012 12:55:42PM 36 points [-]

On counter-signaling, how not to do:

US police investigated a parked car with a personalized plate reading "SMUGLER". They found the vehicle, packed with 24 lb (11 kg) of narcotics, parked near the Canadian border at a hotel named "The Smugglers' Inn." Police believed the trafficker thought that being so obvious would deter the authorities.

-- The Irish Independent, "News In Brief"

Comment author: XFrequentist 24 March 2012 04:01:57AM *  10 points [-]

Questions from DAGGRE with a current estimate between 10 and 90%:

Will Ayman al-Zawahiri still be recognized as the leader of al-Qaeda as of 31 december 2012?

Will there be an official announcement of a new sovereign debt restructuring program for Greece before June 1, 2012?

Will the forces of Alassane Outtara defeat the forces of Laurent Gbagbo in the Ivory Coast before 1 December 2012?

Will Croatia’s GDP grow more than 0 percent in calendar 2012?

Will €1 Euro buy less than $1.20 Australian dollars at any point before 1 January 2013?

Will there will be a 50%-effective malaria vaccine available for general use before 2015?

Comment author: Yvain 24 March 2012 01:57:22PM 5 points [-]

These would all make decent examples, but I'm surprised by how many posts here have concentrated solely on future events (which are obviously uncertain). I'm curious whether there are any questions about the past or present that fit these criteria.

Comment author: juliawise 23 March 2012 08:21:28PM 4 points [-]

I wrote a post with your Modest Proposal as a jumping-off point.

Comment author: Yvain 24 March 2012 12:30:00AM 1 point [-]

Thanks for the link, and I agree with pretty much all of what you said.

Comment author: Locke 23 March 2012 04:37:51PM 0 points [-]

The Space Program? There are reasonable people who say it's vital for the future good of our species, and reasonable people who say it's impractical and unnecessary, at least for the time being.

Comment author: Yvain 23 March 2012 04:45:37PM 7 points [-]

Sounds more like a value question to me.

Nonmindkilling open questions

22 Yvain 23 March 2012 04:23PM

When I explain to people how beliefs should be expressed in probabilities, I would like to use an example like "Consider X. Lots of intelligent people believe X, but lots of equally intelligent people believe not-X. It would be ridiculous to say you are 100% sure either way, so even if you have a strong opinion about X, you should express your belief as a probability."

Trouble is, I'm having a hard time thinking of an example to plug into X. For an example to work, it would need the following properties:

Factual question. So no value-laden questions like "Is abortion morally acceptable?" or counterfactual questions like "Would the US have won a war with the Soviet Union in 1960?"

Popular and important question. The average person should be aware it's an issue and care about the answer. So no "My aunt's middle name is Gladys" or "P = NP."

High uncertainty. Reasonable people should be reluctant to give a probability >90% or <10%

No opportunity to gain status by signaling overwhelming support for one side. So cryonics is out, because it's too easy to say "That's stupid, I'm 100% sure cryonics won't work and no intelligent person could believe it." I'm assuming in any debate where you can gain free status by assigning crazy high probabilities to the "responsible" position, people will do just that - so no psi effects, Kennedy assassination, or anything in that cluster. I need a question with no well-accepted "responsible" position.

Minimal mindkilling effect. My previous go-to example has been global warming, but I keep encountering people who say that global warming 100% for sure exists and the only people who could possibly doubt it are oil company shills. Or if I were to try the existence of God, I predict half the population would say it's 100% certain God exists, and the other half would say the opposite.

So what are the important questions that average (or somewhat-above-average) people will likely agree are complicated open questions where both sides have good points? And if there aren't many such questions, what does that say about us?

Comment author: Raemon 21 March 2012 09:53:03PM *  7 points [-]

I have a similar problem with contest-labor. I have less of a problem with it for non-profits. But my reasoning is actually particularly relevant to an organization that is (among other things), promoting rationality. (You could argue that it is either more or less concerning, given your pool of volunteers' propensity for rationality)

My problem with contest labor is that it exploits people's probability biases. They see "I could get $1000!". They don't see "the expected value for this labor is about $1.00/hour" (or less). Which is usually the case (especially for stuff like logo design). I don't know what the expected value is for a contest like this - the numbers are high enough and the people contributing will probably be low enough that it may be a pretty good deal.

I don't think this is wrong per se, but it's Dark Arts-ish. (Approximately as Dark Arts as using anchoring in your advertising, but I'm not sure how bad I consider that in the first place)

(Bonus points to anyone who (for some reason?) has been following my posts closely and can point out inconsistencies in my previous comments on similar issues. I have no justification for the inconsistency)

Comment author: Yvain 21 March 2012 11:24:53PM *  29 points [-]

I trust LWers to do expected utility calculations, but it's actually much worse than this.

We may decide whether or not to enter based on our probabilities about how many other people will enter: if I think many people will enter, I shouldn't waste my time, but if I think few people will enter, I have a good chance and should enter. But we also know all of our potential competitors will be thinking the same, and possibly making predictions with a similar algorithm to ourselves.

That makes this an anticoordination problem similar to the El Farol Bar, which is an especially nasty class of game because it means the majority of people inevitably regret their choice. If we predict few people will enter, then that prediction will make many people enter, and we will regret our prediction. If we predict many people will enter, that prediction will make few people enter, and we will again regret our prediction. As long as our choices are correlated, there's no good option!

The proper response would be to pursue a mixed strategy in which we randomly enter or do not enter the contest based on some calculations and a coin flip, but this would unfairly privilege defectors and be a bit mean to the Singularity Institute, especially if people were to settle on a solution like only one person entering each contest - which might end up optimal since more people entering not only linearly decreases chance of winning, but also increases effort you have to put into your entry, eg if you were the only entrant you could just write a single sentence and win by default.

And you might think: then just let everyone know exactly how many people have entered at any one time. But that turns it into a Malthusianism: people will gain no utility by entering the contest, because utility of entering the contest is a function of how many other people are in the contest, and if there were still utility to be gained, more people would enter the contest until that stopped being true.

(Although this comment isn't entirely serious, I honestly worried about some of these issues before I entered the efficient charity contest and the nutrition contest. And, uh, won both of them, which I guess makes me a dirty rotten defector and totally ruins my point.)

Comment author: Dmytry 17 March 2012 08:36:33PM *  2 points [-]

I've nothing against what you think evo psych is about. Clearly, the sensitivity of receptors at synaptic junctions can be altered in global manner, and clearly, the intensity of any existing process can also be adjusted - similar to how shape of human body differs from that of a chimp, so can the 'shape' of human psyche.

The argument with orthonormal is on an orthogonal subject, ha ha: the evo psych as proposed by Cosmides, Tooby, Pinker, and Symons, whom exactly propose the cognitive modules as complex as organs, in the past 1.8 millions years, and whom even draw organ analogies themselves. I'm aint making up any strawmen here.

edit: Actually i didn't even want to bring up this whole debate, that's why i didn't directly refer to any of that in my original post. I just want to make a sensible argument, people can apply it, and discard the nonsensical variations of evo psych, especially when the evo psychologists themselves make organ analogies. I'd rather attack the fallacious lines of reasoning, than specific arguments.

Comment author: Yvain 17 March 2012 08:49:35PM 3 points [-]

Oh, okay.

Are you sure they believe what you think they believe? I mean, obviously all animals have sexual behaviors, many of them very complicated and different from one another, so it would be pretty nonsensical to say humans were the first animal with a "sex behavior module" in their brains. I can't imagine someone like Cosmides & Tooby would make that mistake

Language seems like an easier mistake to make, but I agree with you that it's a modification and upscaling of existing organs rather than anything new; primates seem to have Broca's and Wernicke's areas in about the same place we do doing about the same sort of thing.

Comment author: Dmytry 17 March 2012 04:41:07PM *  2 points [-]

http://en.wikipedia.org/wiki/Criticism_of_evolutionary_psychology#Testability

In another thread, you linked to some book, I forgot the name, there was critique from another prominent guy saying it's not how mind works. Clearly there is critique.

Do you have an explanation why do you expect considerably more complex changes to wiring in the brain, than to gross morphology, i.e. shape of organism? I honestly just don't see why it would be more common for evolution to hardwire a reflex than to make a new organ. edit: Okay, for the organs there is the argument that the existing organs are pretty damn good. Still, there's plenty of opportunity for improving e.g. human locomotion, and pressure, too, and one can clearly see how slow did the bones change shapes.

edit: ahh, now i have a great question: Why would it be so much more common to evolve some complex but hard to separate from culture psychology, than it was to evolve verifiable, simple, straightforward hardwired reflexes? There's not a single well defined, agreed upon reflex i can think of that humans have, which chimps lack. There's a lot of evo-psych stuff that is allegedly unique to us humans, evolved during our hunter gatherer times.

I think this really should nail it down WRT plausibility of evo-psych. It propositions big number of very complex psychological adaptations, over the time when no straightforward, agreed upon hard wired reflexes evolved. Not just gross morphology. Anything well identifiable at which we can look and say - okay, chimps don't have this innate reflex - and agree.

Comment author: Yvain 17 March 2012 08:30:54PM *  6 points [-]

So I have two problems with your response here.

First, I don't think the really interesting evo psych is about the difference between humans and chimps. The most interesting evo psych is about ideas like coalition politics, mating behavior, and emotions which we share with chimps and with many other primates as well.

(this also brings up the related points about the major "psychological" differences between different primates, or between chimps and bonobos; these seem to clearly exist, but are hard to attribute to culture; if chimp/bonobo differences can be attributed to evo psych, why not chimp/human differences?)

The more fundamental problem I have with your response is that it seems to be placing the burden of proof on Orthonormal by asking "Given such minor anatomical differences between chimps and humans, why are you expecting huge brain differences equivalent to completely new organs?" But who's positing huge brain differences equivalent to completely new organs? Given that there are hundreds of small anatomical differences between chimps and humans, I would almost want to throw the burden of proof back at you and ask "Given all the anatomical differences between chimps and humans, why are you expecting there to be zero mental differences at all except those related to scale?"

It may be that we are thinking of different things when we say "evolutionary psychology". I agree that there's no specific novel human brain module responsible for (let's say) religion: that would be equivalent to evolving a new organ. But could the brain modules handling sex, which certainly exist in other animals and chimps, have a slight difference in humans which explains why we're more naturally monogamous or polygamous or whatever the theory is nowadays? Sure. And that's what I think evo psych is about, more than it's about saying "We must have evolved a specific religion module in the last million years!"

Comment author: XiXiDu 17 March 2012 12:51:53PM 0 points [-]

Yvain, the whole point of the above post was to show that you can't estimate the probability and magnitude of the advantage an AI will have if you are using something that is as vague as 'intelligence'.

Just as it is acceptable to say "General Lee won the battle because he was intelligent", so it is acceptable to say "The AI would conquer Rome because it was intelligent".

No, it is not. Because it is seldom intelligence that makes people win. Very high IQ is not correlated with might and power. Neither does evolution favor intelligence.

Comment author: Yvain 17 March 2012 01:15:35PM 5 points [-]

Neither does evolution favor intelligence.

Evolution doesn't universally favor any one trait (see the Sequences). It favors whatever is most useful in a particular situation. That having been said, intelligence has proven pretty useful so far; humans seem more evolutionarily successful than chimps, and I'd drag in intelligence rather than bipedalism or hairlessness or whatever to explain that.

More importantly, when you say high IQ isn't correlated with might and power, I think you're thinking of miniscule, silly differences like the difference between the village idiot and Albert Einstein (see the Sequences). Let's think more "human vs. dog". In a battle between two armies, the one led by a human and the other by a dog, the human will win every time. Given enough time to plan and enough access to the products of other humans, a human can win any conceivable contest against a dog, even if that dog has equal time to plan and equal access the products of other dogs. Very high IQ is totally correlated with might and power when "very high IQ" means "IQ of at least five digits". (see the Sequences )

Comment author: XiXiDu 17 March 2012 12:25:50PM *  1 point [-]

(Addendum to my other comment)

Here is why I believe that reading the Sequences might not be worth the effort:

1) According to your survey, 38.5% of all people have read at least 75% of the Sequences yet only 16.5% think that unfriendly AI is the most fearsome existential risk.

2) The following (smart) people have read the Sequences, and more, but do not agree about risks from AI:

  • Robin Hanson
  • Katja Grace (who has been a visiting fellow)
  • John Baez (who interviews Eliezer Yudkowsky)
  • Holden Karnofsky
  • Ben Goertzel
Comment author: Yvain 17 March 2012 12:54:32PM 2 points [-]

I hope I didn't claim the Sequences, or any argument were 100% effective in changing the mind of every single person who read them.

Also, Ben Goertzel has read all the Sequences? That makes that recent conversation with Luke kind of sad.

Comment author: JenniferRM 17 March 2012 02:10:35AM *  4 points [-]

Upvoted and hopefully answered :-)

Specifically, I think you might be missing the halo effect, the fundamental attribution error, survivorship bias, and strategic signalling to gain access to power, influence, and money.

What is the nature of the property that the general would have a 93% chance of having? Is it a property you'd hypothesize was shared by about 7% of all humans in history? Is it shared by 7% of extant generals? What if the internal details of the property you hypothesize is being revealed are such that no general actually has it, even though some general always wins each battle? How would you distinguish between these outcomes? How many real full scale battles are necessary and how expensive are they to run to push P(at least one general has the trait) and P(a specific general has the trait|at least one general has the trait) close to 1 or 0?

XiXiDu titled his article "The Futility Of Intelligence". What I'm proposing is something more like "The Use And Abuse Of Appearances Of General Intelligence, And What Remains Of The Theory Of General Intelligence After Subtracting Out This Noise". I think that there is something left, but I suspect it isn't as magically powerful or generic as is sometimes assumed, especially around these parts. You have discussed similar themes in the past in less mechanistic and more personal, friendly, humanized, and generally better written forms :-)

This point is consonant with ryjm's sibling comment but if my suspicions stand then the implications are not simply "subtle and not incredibly useful" but have concrete personal implications (it suggests studying important domains is more important than studying abstractions about how to study, unless abstraction+domain is faster to acquire than the domain itself, and abstraction+abstraction+domain faces similar constraints (which is again not a particularly original insight)). The same suspicion has application to political discourse and dynamics where it suggests that claims of generic capacity are frequently false, except when precise mechanisms are spelled out, as with market pricing as a reasonably robust method for coordinating complex behaviors to achieve outcomes no individual could achieve on their own.

A roughly analogous issue comes up in the selection of "actively managed" investment funds. All of them charge something for their cognitive labor and some of them actually add value thereby, but a lot of it is just survivorship bias and investor gullibility. "Past performance is no guarantee of future results." Companies in that industry will regularly create new investment funds, run them for a while, and put the "funds that have survived with the best results so far" on their investment brochures while keeping their other investment funds in the background where stinkers can be quietly culled. Its a good trick for extracting rent from marks, but it's not the sort of thing that would be done if there was solid and simple evidence of a "real" alpha that investors could pay attention to as a useful and generic predictor of future success without knowing much about the context.

I have a strong suspicion, and I'd love this hunch to be proved wrong, that there's mostly no free lunches when it comes to epistemology. Being smart about one investment regime is not the same as being smart about another investment regime. Being a general and playing chess have relatively little cross-applicable knowledge. Being good at chess has relatively little in common with being good at the abstractions of game theory.

With this claim (which I'm not entirely sure of because its very abstract and hard to ground in observables) I'm not saying that AGI that implements something like "general learning ability in silicon and steel" wouldn't be amazing or socially transformative, I'm not saying that extreme rationality is worthless, its more like I'm claiming that its not magic, with a sub-claim that sometimes some people seem to speak (and act?) as though they think it might be magic. Like they can hand-wave the details because they've posited "being smarter" as an ontologically basic property rather than as a summary for having nailed down many details in a functional whole. If you adopt an implementation perspective, then the summary evaporates because the details are what remain before you to manipulate.

Comment author: Yvain 17 March 2012 11:57:41AM *  7 points [-]

So I'm interpreting your point as being "What if what we think of when we say 'general intelligence' isn't really all that useful in different domains, but we keep treating it as if it were the kind of thing that could constantly win battles or conquer Rome or whatever?" Perhaps then it was a mistake to talk about generals in battle, as your theory is that there may be an especially victorious general, but his fortune may be due more to some specific skill at tactics than his general intelligence?

I guess my belief in the utility of general intelligence (you cited an article of mine arguing against huge gains from technical rationality, which I consider very different; here I'm talking about pure IQ) would come from a comparison with subnormal intelligence. A dog would make a terrible general. To decreasing degrees, so too would a chimp, a five year old child, a person with Down's Syndrome, and most likely a healthy person with an IQ of 75. These animals and people would also, more likely than not, be terrible chess players, mathematicians, writers, politicians, and chefs.

This is true regardless of domain-specific training: you can read von Clausewitz's On War to a dog and it will just sit there, wagging its tail. You can read it to a person with IQ 75, and most of the more complicated concepts will be lost. Maybe reading On War would allow a person with a few dozen IQ point handicap to win, but it's not going to make a difference across a gulf the size of the one between dogs and humans.

Humans certainly didn't evolve a separate chess playing module, or a separate submarine tactics module, so we attribute our being able to wipe the floor with dogs and apes in chess or submarine warfare to some kind of "high general intelligence" we have and they don't.

So to me, belief in a general intelligence that could give AIs an advantage is just the antiprediction that the things that kept being true up until about IQ 100 still continue to be true after that bar. Just as we expect a human to be able to beat a dog at chess (even if we could get the dog to move pieces with its nose or something), and we would use the word "intelligence" to explain why, so I would expect Omega to be able to beat a human for the same reason.

Is that a little closer to the point of your objection?

Comment author: Yvain 17 March 2012 12:42:19AM 6 points [-]

Now I'm curious why, as far as I know, no animal has ever evolved an eye in the back of its head. You mention bipedality and descent from trees, but why wouldn't a gazelle or a fly or a small fish get an advantage from one?

Schelling fences on slippery slopes

179 Yvain 16 March 2012 11:44PM

Slippery slopes are themselves a slippery concept. Imagine trying to explain them to an alien:

"Well, we right-thinking people are quite sure that the Holocaust happened, so banning Holocaust denial would shut up some crackpots and improve the discourse. But it's one step on the road to things like banning unpopular political positions or religions, and we right-thinking people oppose that, so we won't ban Holocaust denial."

And the alien might well respond: "But you could just ban Holocaust denial, but not ban unpopular political positions or religions. Then you right-thinking people get the thing you want, but not the thing you don't want."

This post is about some of the replies you might give the alien.

Abandoning the Power of Choice

This is the boring one without any philosophical insight that gets mentioned only for completeness' sake. In this reply, giving up a certain point risks losing the ability to decide whether or not to give up other points.

For example, if people gave up the right to privacy and allowed the government to monitor all phone calls, online communications, and public places, then if someone launched a military coup, it would be very difficult to resist them because there would be no way to secretly organize a rebellion. This is also brought up in arguments about gun control a lot.

I'm not sure this is properly thought of as a slippery slope argument at all. It seems to be a more straightforward "Don't give up useful tools for fighting tyranny" argument.

The Legend of Murder-Gandhi

Previously on Less Wrong's The Adventures of Murder-Gandhi: Gandhi is offered a pill that will turn him into an unstoppable murderer. He refuses to take it, because in his current incarnation as a pacifist, he doesn't want others to die, and he knows that would be a consequence of taking the pill. Even if we offered him $1 million to take the pill, his abhorrence of violence would lead him to refuse.

But suppose we offered Gandhi $1 million to take a different pill: one which would decrease his reluctance to murder by 1%. This sounds like a pretty good deal. Even a person with 1% less reluctance to murder than Gandhi is still pretty pacifist and not likely to go killing anybody. And he could donate the money to his favorite charity and perhaps save some lives. Gandhi accepts the offer.

Now we iterate the process: every time Gandhi takes the 1%-more-likely-to-murder-pill, we offer him another $1 million to take the same pill again.

Maybe original Gandhi, upon sober contemplation, would decide to accept $5 million to become 5% less reluctant to murder. Maybe 95% of his original pacifism is the only level at which he can be absolutely sure that he will still pursue his pacifist ideals.

Unfortunately, original Gandhi isn't the one making the choice of whether or not to take the 6th pill. 95%-Gandhi is. And 95% Gandhi doesn't care quite as much about pacifism as original Gandhi did. He still doesn't want to become a murderer, but it wouldn't be a disaster if he were just 90% as reluctant as original Gandhi, that stuck-up goody-goody.

What if there were a general principle that each Gandhi was comfortable with Gandhis 5% more murderous than himself, but no more? Original Gandhi would start taking the pills, hoping to get down to 95%, but 95%-Gandhi would start taking five more, hoping to get down to 90%, and so on until he's rampaging through the streets of Delhi, killing everything in sight.

Now we're tempted to say Gandhi shouldn't even take the first pill. But this also seems odd. Are we really saying Gandhi shouldn't take what's basically a free million dollars to turn himself into 99%-Gandhi, who might well be nearly indistinguishable in his actions from the original?

Maybe Gandhi's best option is to "fence off" an area of the slippery slope by establishing a Schelling point - an arbitrary point that takes on special value as a dividing line. If he can hold himself to the precommitment, he can maximize his winnings. For example, original Gandhi could swear a mighty oath to take only five pills - or if he didn't trust even his own legendary virtue, he could give all his most valuable possessions to a friend and tell the friend to destroy them if he took more than five pills. This would commit his future self to stick to the 95% boundary (even though that future self is itching to try to the same precommitment strategy to stick to its own 90% boundary).

Real slippery slopes will resemble this example if, each time we change the rules, we also end up changing our opinion about how the rules should be changed. For example, I think the Catholic Church may be working off a theory of "If we give up this traditional practice, people will lose respect for tradition and want to give up even more traditional practices, and so on."

Slippery Hyperbolic Discounting

One evening, I start playing Sid Meier's Civilization (IV, if you're wondering - V is terrible). I have work tomorrow, so I want to stop and go to sleep by midnight.

At midnight, I consider my alternatives. For the moment, I feel an urge to keep playing Civilization. But I know I'll be miserable tomorrow if I haven't gotten enough sleep. Being a hyperbolic discounter, I value the next ten minutes a lot, but after that the curve becomes pretty flat and maybe I don't value 12:20 much more than I value the next morning at work. Ten minutes' sleep here or there doesn't make any difference. So I say: "I will play Civilization for ten minutes - 'just one more turn' - and then I will go to bed."

Time passes. It is now 12:10. Still being a hyperbolic discounter, I value the next ten minutes a lot, and subsequent times much less. And so I say: I will play until 12:20, ten minutes sleep here or there not making much difference, and then sleep.

And so on until my empire bestrides the globe and the rising sun peeps through my windows.

This is pretty much the same process described above with Murder-Gandhi except that here the role of the value-changing pill is played by time and my own tendency to discount hyperbolically.

The solution is the same. If I consider the problem early in the evening, I can precommit to midnight as a nice round number that makes a good Schelling point. Then, when deciding whether or not to play after midnight, I can treat my decision not as "Midnight or 12:10" - because 12:10 will always win that particular race - but as "Midnight or abandoning the only credible Schelling point and probably playing all night", which will be sufficient to scare me into turning off the computer.

(if I consider the problem at 12:01, I may be able to precommit to 12:10 if I am especially good at precommitments, but it's not a very natural Schelling point and it might be easier to say something like "as soon as I finish this turn" or "as soon as I discover this technology").

Coalitions of Resistance

Suppose you are a Zoroastrian, along with 1% of the population. In fact, along with Zoroastrianism your country has fifty other small religions, each with 1% of the population. 49% of your countrymen are atheist, and hate religion with a passion.

You hear that the government is considering banning the Taoists, who comprise 1% of the population. You've never liked the Taoists, vile doubters of the light of Ahura Mazda that they are, so you go along with this. When you hear the government wants to ban the Sikhs and Jains, you take the same tack.

But now you are in the unfortunate situation described by Martin Niemoller:

First they came for the socialists, and I did not speak out, because I was not a socialist.
Then they came for the trade unionists, and I did not speak out, because I was not a trade unionist.
Then they came for the Jews, and I did not speak out, because I was not a Jew.
Then they came for me, but we had already abandoned the only defensible Schelling point

With the banned Taoists, Sikhs, and Jains no longer invested in the outcome, the 49% atheist population has enough clout to ban Zoroastrianism and anyone else they want to ban. The better strategy would have been to have all fifty-one small religions form a coalition to defend one another's right to exist. In this toy model, they could have done so in an ecumenial congress, or some other literal strategy meeting.

But in the real world, there aren't fifty-one well-delineated religions. There are billions of people, each with their own set of opinions to defend. It would be impractical for everyone to physically coordinate, so they have to rely on Schelling points.

In the original example with the alien, I cheated by using the phrase "right-thinking people". In reality, figuring out who qualifies to join the Right-Thinking People Club is half the battle, and everyone's likely to have a different opinion on it. So far, the practical solution to the coordination problem, the "only defensible Schelling point", has been to just have everyone agree to defend everyone else without worrying whether they're right-thinking or not, and this is easier than trying to coordinate room for exceptions like Holocaust deniers. Give up on the Holocaust deniers, and no one else can be sure what other Schelling point you've committed to, if any...

...unless they can. In parts of Europe, they've banned Holocaust denial for years and everyone's been totally okay with it. There are also a host of other well-respected exceptions to free speech, like shouting "fire" in a crowded theater. Presumably, these exemptions are protected by tradition, so that they have become new Schelling points there, or are else so obvious that everyone except Holocaust deniers is willing to allow a special Holocaust denial exception without worrying it will impact their own case.

Summary

Slippery slopes legitimately exist wherever a policy not only affects the world directly, but affects people's willingness or ability to oppose future policies. Slippery slopes can sometimes be avoided by establishing a "Schelling fence" - a Schelling point that the various interest groups involved - or yourself across different values and times - make a credible precommitment to defend.

Comment author: JenniferRM 16 March 2012 05:06:25PM *  7 points [-]

I think XiXiDu's arguments are an over-reach but I think they contain an element of truth that is functionally important and that you're not giving him credit for. The place in your response where the insight is lurking seems to me like this bit:

The same is true of "General Lee won his battles because he was intelligent". It distinguishes the case in which he was a tactical genius from the case where he just had overwhelming numbers, or was very lucky, or had superior technology.

I understand what I'd expect to see if a victory came from overwhelming numbers or superior technology. Overwhelming numbers would be easy to quantify and plug into a very coarse grained victory predicting battled simulation. Superior technology would be trickier, but could probably be accounted for in terms of higher rate of fire, faster supply lines, better communication, and more detailed simulation of these and similar technological elements.

However, I'd be hard pressed to quantify "tactical ability" or "luckiness" other than by pitting generals against each other over and over and seeing who won, kinda the way strength in the games of go and chess are assessed, except in the one case attributing the outcomes to brains and in the other to noise. Perhaps I could simulate everything really precisely, including their minds and give the one with more "tactical ability" more clock cycles and RAM? But it would be tricky. Some real world algorithms actually do better with fewer clock cycles, because they over fit the training data if you give them enough rope to hang themselves with.

Fiction generally communicates more vividly when the elements of the story shown to the reader rather than told to the reader. "His brow furrowed and his eyes flicked back and forth." vs "He wasn't good at hiding it when the surprise made him suspicious." Telling communicates a theory about what happened without bothering with the evidence; showing gives you the evidence. When writers of fiction try to show superintelligence, they run into a problem that can be solved in various ways, but to some degree it amounts to operationalizations of the hypothesis that a particular agent in the story is really smart. I've been collecting fiction and "non-fiction about the fiction" in this area for a while and one of them seems appropriate here.

Charlie Stross has written much about superintelligence, so he's spent a fair amount of time trying to figure out what you'd expect to observe, and one of his theories on this score has been floating around for a while:

On a related note, I once heard Stross talk about how to write superintelligences, and he gave an illustrative example, paraphrased: "When I need to take my cat to the vet, I bring out the cat carrier. The cat knows what this means, and then runs for the cat door. He is then very surprised to discover that it's closed and locked. To him, these two events are totally coincidental. I think that it's easy to write a story about a superintelligence: just have any humans that try to act against it constantly surprised by apparent coincidences that turn out to all have been the superintelligence's fault in the end."

Suppose I'm trying to thwart a superintelligence, as an exercise, and I end up tripping on a rock at the key moment. Afterwards, my expectations is that the superintelligence hasn't, for example, saturated my environment with rocks with hidden mechanisms that can shift on command, causing me to trip when it's observed to be required because that would be wasteful and dumb. Instead I expect it to have placed a handful of such rocks in just the right places, and to be able to explain the basis for placing the rocks in those particular locations, and I expect its explanation to make sense in retrospect. It might take me three months to read and ponder and verify the explanation, but then at the end I should say "yeah, that was all just common sense and keen observation, iterated over and over".

What I expect, in other words, is that there exists predictable structure and pattern in the world that I'm just not noticing right now, but that I hypothesize someone else could notice if they were integrating more information, using better modeling, at faster clock cycles, in a more goal directed fashion. The hypothesis of "effective superintelligence" thus pre-supposes that there's this "low hanging idea fruit" out of our human reach for really banal reasons like having 7 plus or minus 2 working memory registers (rather than 15), or living 70 years rather than 140 (and thus having to curtail learning and go to seed equivalently earlier in our life cycle).

Another way to explain the issue in more vivid terms, closer to near mode... Imagine that some general wins four seemingly even battles in a row and gives a lot of post hoc reasoning about how and why he won. "Yes," he concludes while being quite tall and quite male and buffing his nails against his chest, "I guess I'm just a genius. Maybe you should make me president given how awesome I am." But one in sixteen equivalently competent generals would appear the same. And post hoc explanations are disturbingly easy to accept. Maybe the general just got lucky? (The luck hypothesis being Fermi's side in his famous conversation with Groves.) Or maybe he actually is particularly skilled but in a domain limited way? How would you tell whether it was luck, or not? How would you tell if it was domain agnostic, or not? Especially, how would you tell without having domain expertise yourself?

Imagine Omega created a duplicate earth exactly like ours (geography, ecology, etc) except humans were genetically adjusted so they had 15 working memory registers and lived to 140, and they attended school till they were 45 as a casual matter so they had to-us-encyclopedic knowledge of "the universe in general and how it works". Suppose one of them is transported here at age 50 (looking like a 25 year old). After they got over the squalor and amorality and the fact that we all resented them for being better than us, they might take over the world. Sure... that could be. Or instead they might "flirt with leftist causes and be briefly in the news after being arrested for involvement in a socialist rally that turned into a riot. After that, nothing."*.

Can you imagine a super intelligence whose long term result is "after that, nothing"? If value of information calculations exhibit diminishing marginal utility in the general case, this is precisely what I would expect. The early swift and meaningful rise happens when truly valuable information is being acquired faster than normal, but once the prodigy had the basic gist of the world in their head they would be pretty similar to everyone else with the same core knowledge. And yet if I saw someone getting better than normal outcomes in some domain that I didn't understand very well "superior intelligence" would be a tempting hypothesis. "Luck" or "intelligence"? How do you tell the difference from a distance? I don't have an answer but I think that generic falsification criteria to answer this question in advance in a domain agnostic way would be very valuable.

Comment author: Yvain 16 March 2012 06:34:13PM 6 points [-]

How would you tell whether it was luck, or not? How would you tell if it was domain agnostic, or not? Especially, how would you tell without having domain expertise yourself?

Given that all knowledge is probabilistic, it seems to me that I should believe there's a 93% chance he's skilled and a 7% chance he's lucky assuming equal prior probability. You could probably up your certainty a little by investigating whether other generals thought his tactics were brilliant or stupid, whether he has related skills like being good at chess and wargames, what his grades were at West Point, whether he conformed to military science's conventional wisdom as to the best tactics in a situation, and whether the narrative of the battles contain any obvious element of luck like "...and then a meteorite struck the enemy command center". Since luck is just probability, it shouldn't be too hard to calculate, and if that and intelligence are your only options, the opposite probability is intelligence.

That seems too simple an answer. Tell me what I'm missing.

Comment author: Yvain 15 March 2012 06:10:44PM *  39 points [-]

Compare "intelligent" to "fast".

I say "The cheetah will win the race, because it is very fast."

This has some explanatory power. It can distinguish between various reasons a cheetah might win a race: maybe it had a head start, or its competitors weren't trying very hard, or it cheeted. Once we say "The cheetah won the race because it was fast" we know more than we did before.

The same is true of "General Lee won his battles because he was intelligent". It distinguishes the case in which he was a tactical genius from the case where he just had overwhelming numbers, or was very lucky, or had superior technology. So "intelligent" is totally meaningful here.

None of these are a lowest-level explanation. We can further explain a cheetah's fast-ness by talking about its limb musculature and its metabolism and so on, and we can further explain General Lee's intelligence by talking about the synaptic connections in his brain (probably). But we don't always need the lowest possible level of explanation; no one gets angry because we don't explain World War I by starting with "So, there were these electrons in Gavrilio Princip's brain that got acted upon by the electromagnetic force..."

A Mysterious Explanation isn't just any time you use a non-lowest-level explanatory word. It's when you explain something on one level by referring to something on the same level.

I have lost track of how many times I have heard people say, "an artificial general intelligence would have a genuine intelligence advantage" as if that explained its advantage.

Just as it is acceptable to say "General Lee won the battle because he was intelligent", so it is acceptable to say "The AI would conquer Rome because it was intelligent".

(just as it is acceptable to say "cavalry has an advantage over artillery because it is fast")

In fact, in the context of the quote, we were talking about the difference between a random modern human trying to take over Rome, and an AI trying to take over modern civilization. The modern human's advantage would be in technology and foreknowledge (as if General Lee won his battles by having plasma rifles and knowing all the North's moves in advance even though he wasn't that good a tactician); the AI might have those advantages, but also be more intelligent.

How would you take over Rome?

25 Yvain 14 March 2012 04:24PM

A recent discussion post has compared the difficulty of an AI destroying modern human civilization to that of a modern human taking over the Roman Empire, with the implication that it is impossible.

The analogy has a few problems: first, modern humans don't have much greater raw intelligence than the Romans, only a bit more knowledge and tools; an AI would have a genuine intelligence advantage. Second, a high-tech civilization like ours offers many more ways for a genius to cause chaos than existed in classical Rome: it's more plausible that you can throw a few existing technologies together to create a superweapon than that Ptolemy could have done likewise, and there's no ancient Roman equivalent to hacking a nuclear launch system.

But taking over ancient Rome might serve as an interesting upper bound on the difficulty of an AI taking over modern civilization. And it's a theme of Harry Potter and the Methods of Rationality that rationalists should be able to come up with creative solutions to seemingly hard problems. So if Professor Quirrell offered it as an extra credit assignment, how would you take over Rome?

Here are the rules:

- You are thrown back in time to the year 1 AD. You can choose to arrive anywhere in the world, but your method of arrival cannot itself give an advantage (you can't appear in a flash of light in the middle of a religious ritual or anything).

- You do not start with Roman citizenship or any other legal record of your existence.

- You keep your original physical characteristics, including sex, height, and fitness. You will appear in period-appropriate dress of your choosing, and can't carry any artifacts with you. You may start with enough money to live a patrician lifestyle for a year.

- You are intellectually near-perfect. You know all human knowledge as of 2012. You speak fluent Latin (and all other languages of the day) and can orate as eloquently as Cicero or Demosthenes. You are a tactical genius of the order of Caesar and Napoleon. And you have infinite willpower and goal-directedness: aside from human necessities like sleep or food, you need never rest.

- You win if you either become Roman Emperor (and are acknowledged as such by most Romans), or if a state you control conquers the city of Rome. You lose if you die, of old age or otherwise, before completing either goal.

Comment author: Yvain 13 March 2012 10:28:29PM 19 points [-]

This is a good post, but it would be super valuable if you could explain the more advanced decision theories and the current problems people are working on as clearly as you explained the basics here.

Comment author: Yvain 13 March 2012 10:01:38PM *  29 points [-]

Imagine a group of 100 world-renowned scientists and military strategists. Could such a group easily wipe away the Roman empire when beamed back in time?

Imagine a group of 530 Spaniards...

At the risk of confirming every negative stereotype RationalWiki and the like have of us...have you read the Sequences? I'm reluctant to write a full response to this, but I think large parts of the Sequences were written to address some of these ideas.

Comment author: Yvain 08 March 2012 06:17:34PM 4 points [-]

Seen here:

Harvard psychologist and APS Fellow and Charter Member Ellen Langer observed similar rule-based behavior in a typical office setting. She had researchers ask if they could cut in line to use a copy machine. When they simply said, “Excuse me, may I use the copy machine?”, only 60 percent of the subjects complied. When the researchers gave a reason — “Excuse me, may I use the copy machine because I’m in a rush?” — 94 percent said yes. Langer tested this one more time with the phrase, “Excuse me, may I use the copy machine because I need to make some copies,” and again 93 percent of respondents agreed — despite the fact that “I need to make some copies” is not really a very good reason for cutting in line. The way Langer and Cialdini describe it, people hear the word “because” and assume that there is a good reason. That is to say, the word “because” is a shortcut people use to distinguish between good arguments and bad. "

Comment author: moridinamael 08 March 2012 12:02:58AM 4 points [-]

I have two separate objections.

1) The first objection, which I hint at in my original post, is that the selection of a reference class of human beings seems to be selected specifically to make whatever point the anthropic reasoner is trying to make. Why don't I have an equal likelihood of being

  • any available 1.4 kg oblate lump of matter, including rocks and large jellyfish, or
  • any animal possessing more than 100 neurons, or
  • any human being who is aware of the concept of anthropic reasoning, or
  • any sentient being capable of self-awareness ...

Obviously each of these reference classes generate completely different answers. You would guess that, e.g. for each reference class,

  • most 1.4 kg lumps of matter must be brains; or
  • that most animals possessing more than 100 neurons must be humans; or
  • that the concept of anthropic reasoning or the human race dies out pretty soon; or
  • that there must not be any other self-conscious life forms in the universe.

In other words, there is no good reason to pretend that human consciousness works anything like a virtual-parallel-Earth device rather than being like any other conceivable reference class.

2) The second objection is, again, that the existence of "observers" in the first place is an illusion. You have a 100% likelihood of being you because you are identically you (minus some small increment allowing for insanity, etc.). We aren't souls injected into bodies from heaven. We are matter that thinks it has identity.

Comment author: Yvain 08 March 2012 12:25:55AM *  3 points [-]

Let me start with the second question, since I think I have a little more of a clue how to answer it.

Anthropics doesn't really rely on you being you. You being you is just...I guess I could call it a convenient Schelling point. We've got to choose someone to do anthropics on. Suppose we chose Genghis Khan. We could say that Genghis was a conquering warlord, so therefore most people throughout history must be conquering warlords. But this would fail, because the only reason we selected Genghis Khan to begin with was that he was a conquering warlord.

But this selection bias is inherent in anything we try. If we were to deliberately select some random peasant from Khan's era to do anthropics on, so that we avoided that first bias, we would be biasing ourselves towards peasants, biasing ourselves toward people who weren't important, biasing ourselves towards people from the past, and biasing ourselves to Earthlings.

The fact that you are necessarily you is part of why anthropics works. If we were souls who chose bodies at the moment of birth, I couldn't condition on my own existence in 2012, because my soul might have been really excited at the prospect to go into one of those super-rare presingularity bodies. As it is, I know I have no selection bias in selecting myself with my specific characteristics, because I did not select myself or my personal characteristics. So I can look at those characteristics - white human male born in 1984 - and consider them a random sample of the characteristics of all people everywhere and everywhen, and do anthropics on them. The fact that the person who is a suitable random sample for anthropics also happens to be me is probably overemphasized, but I don't think it's that important.

And I think this also goes part of the way to solving your first objection. We can't do anthropics on my brain as a representative sample of all 1.4 kg lumps of matter, because it's getting selected specifically as a 1.4 kg lump of matter that is especially interesting to me - most 1.4 kg lumps of matter were disqualified before they even had a chance to be the one we're doing anthropics on.

I admit I am still quite confused on how this works in more complicated scenarios - see this post for the same argument in the opposite direction.

Comment author: moridinamael 07 March 2012 09:33:10PM 1 point [-]

I have been extremely confused about why anthropics is treated the way it is, I am asking for clarification. I will first explain my current position:

There is no such thing as an "observer." The problem with anthropic reasoning is the same as the problem with the Copenhagen interpretation of quantum mechanics, namely, it invokes non-real magical properties of human minds which I thought we had safely dissolved at this point.

Take your pick of the various anthropic assumptions. Each of them treats the "observer" as some kind of ontologically granular spirit, and that there are somehow equal odds that you might have found your observer-spirit situated in the head of any given human throughout history and into the future. (Setting aside the unjustified assertion that my observer-spirit can inhabit human skulls but not animal skulls or distant alien skulls.) This sense that we have of ourselves as embodied discrete entities is an illusion, a holographic accident of physics.

You could not have been anyone but what you are - namely, one specific dynamic pattern of matter and energy embedded within physics.

If you've lived on an island your whole life with a hundred fellow tribespeople, you would not be right in saying, "There probably aren't any people on any other islands, because if there were a lot of other people, the odds would be really small that I was born on this island." I don't know the name for this error in thinking, but I'm sure it has one.

Perhaps I'm just profoundly misunderstanding the point of anthropics.

Comment author: Yvain 07 March 2012 11:39:25PM *  4 points [-]

You could not have been anyone but what you are - namely, one specific dynamic pattern of matter and energy embedded within physics.

So imagine that post-singularity scientists find evidence of a Parallel Earth, but aren't able to measure or observe it directly. After a bunch of work, they come up with a device that can interact with the parallel world, but only in a very specific way: it connects to the mind of a single intelligent life-form there, and transmits its thoughts and sensations back to you in a sort of "virtual reality". Only problem is they don't know how to aim it: as far as anyone knows, it samples randomly across all space and time when choosing its subject. Also, they only have enough funding to use the device once.

So they aim it and they end up in the body of some guy. The tech level around him seems to be approximately medieval, and he seems to be speaking a Tocharian language. Unfortunately, just then some barbarians show up and kill him, and the device explodes.

What information can we glean about Parallel Earth from this experiment? Well, we know that at least one person spoke Tocharian there in medieval times. But it's unlikely that just one person speaks Tocharian there - if one person spoke Tocharian, and everyone else speaks (let's say) Basque, then it would be hugely improbable that the random device would have chosen the one Tocharian speaker. Because our random sampling device chose a Tocharian speaker, we have some evidence that Tocharian is probably one of the more common languages on Parallel Earth.

We can go even further. Our subject lived in medieval times, we live in post-singularity times and observe a galactic population of five hundred trillion. If Parallel Earth also experiences a singularity with a population of five hundred trillion, and we are genuinely selecting at random from everyone who ever lived, it would be extremely weird for our machine to randomly select one of the (let's say) 500 million people in ancient times as opposed to the 500 trillion people in post-singularity times: in fact, the chances are only (500 mil/500 tril) = (1/1 million). This provides Bayesian evidence that Parallel Earth humanity managed to destroy itself before reaching a singularity.

Anthropics is just the belief that just as I can draw inferences from some machine placing me a randomly selected Parallel Earthling's body, so I can draw inferences from blind luck placing me in a randomly selected Real Earthling's body.

I'm curious to know where you disagree here. Do you think the scientists shouldn't draw inferences based on their device selecting that ancient Tocharian-speaker, or do you think they are justified in their assumptions but present-day anthropic reasoners are not?

In response to How to Fix Science
Comment author: Yvain 03 March 2012 11:53:59PM *  53 points [-]

I only had time to double-check one of the scary links at the top, and I wasn't too impressed with what I found:

In 2010, a careful review showed that published industry-sponsored trials are four times more likely to show positive results than published independent studies, even though the industry-sponsored trials tend to use better experimental designs.

But the careful review you link to claims that studies funded by the industry report 85% positive results, compared to 72% positive by independent organizations and 50% positive by government - which is not what I think of when I hear four times! They also give a lot of reasons to think the difference may be benign: industry tends to do different kinds of studies than independent orgs. The industry studies are mainly Phase III/IV - a part of the approval process where drugs that have already been shown to work in smaller studies are tested on a larger population; the nonprofit and government studies are more often Phase I/II - the first check to see whether a promising new chemical works at all. It makes sense that studies on a drug which has already been found to probably work are more positive than the first studies on a totally new chemical. And the degree to which pharma studies are more likely to be late-phase is greater than the degree to which pharma companies are more likely to show positive results, and the article doesn't give stats comparing like to like! The same review finds with p < .001 that pharma studies are bigger, which again would make them more likely to find a result where one exists.

The only mention of the "4x more likely" number is buried in the Discussion section and cites a completely different study, Lexchin et al.

Lexchin reports an odds ratio of 4, which I think is what your first study meant when they say "industry studies are four times more likely to be positive". Odds ratios have always been one of my least favorite statistical concepts, and I always feel like I'm misunderstanding them somehow, but I don't think "odds ratio of 4" and "four times more likely" are connotatively similar (someone smarter, please back me up on this?!). For example, the largest study in Lexchin's meta-analysis, Yaphe et al, finds that 87% of industry studies are positive versus 65% of independent studies, for an odds ratio of 3.45x. But when I hear something like "X is four times more likely than Y", I think of Y being 20% likely and X being 80% likely; not 65% vs. 87%.

This means Lexchin's results are very very similar to those of the original study you cite, which provides some confirmation that those are probably the true numbers. Lexchin also provides another hypothesis for what's going on. He says that "the research methods of trials sponsored by drug companies is at least as good as that of non-industry funded research and in many cases better", but that along with publication bias, industry fudges the results by comparing their drug to another drug, and then giving the other drug wrong. For example, if your company makes Drug X, you sponsor a study to prove that it's better than Drug Y, but give patients Drug Y at a dose that's too low to do any good (or so high that it produces side effects). Then they conduct that study absolutely perfectly and get the correct result that their drug is better than another drug at the wrong dosage. This doesn't seem like the sort of thing Bayesian statistics could fix; in fact, it sounds like it means study interpretation would require domain-specific medical knowledge; someone who could say "Wait a second, that's not how we usually give penicillin!" I don't know whether this means industry studies that compare their drug against a placebo are more trustworthy.

So, summary. Industry studies seem to hover around 85% positive, non-industry studies around 65%. Part of this is probably because industry studies are more likely to be on drugs that there's already some evidence that they work, and not due to scientific misconduct at all. More of it is due to publication bias and to getting the right answer to a wrong question like "Does this work better than another drug when the other is given improperly?".

Phrases like "Industry studies are four times more likely to show positive results" are connotatively inaccurate and don't support any of these proposals at all, except maybe the one to reduce publication bias.

This reinforces my prejudice that a lot of the literature on how misleading the literature is, is itself among the best examples of how misleading the literature is.

Comment author: Yvain 02 March 2012 04:58:26PM *  42 points [-]

From a simple utilitarian perspective, identifiability is bias. By increasing altruism toward the identifiable victims, it may reduce altruism toward the unidentified ones, who are often the ones most in need of help. On the other hand, it could also increase overall altruism, by making people more willing to incur greater personal costs to help the identifiable victims.

So part of what I think is going on here is that giving to statistical charity is a slippery slope. There is no one number that it's consistent to give: if I give $10 to fight malaria, one could reasonably ask why I didn't give $100; if I give $100, why not $1000; and if $1000, why not every spare cent I make? Usually when we're on a slippery slope like this, we look for a Schelling point, but there are only two good Schelling points here: zero and every spare cent for the rest of your life. Since most people won't donate every spare cent, they stick to "zero". I first realized this when I thought about why I so liked Giving What We Can's philosophy of donating 10% of what you make; it's a powerful suggestion because it provides some number between 0 and 100 which you can reach and then feel good about yourself.

Then identifiable charity succeeds not just because it attaches a face to people, but also because it avoids the slippery slope. If we're told we need to donate to save "baby Jessica", it's very easy to donate exactly as much money as is necessary to help save baby Jessica and then stop. The same is true of natural disasters; if there's an earthquake in Haiti, that means we can donate money to Haiti today but not be under any consistency-related obligations to do so again until the next earthquake. If Haiti is just a horrible impoverished country, then there's no reason to donate now as opposed to any other time, and this is true for all possible "now"s.

Feedback appreciated as I've been planning to make a top-level post about this if I ever get time.

Comment author: Craig_Heldreth 19 February 2012 03:47:07PM 9 points [-]

Here's what it says on the label of mine:

Manganese 2.3 Mg 115%

Anybody want to plug a one pill supplement a middle class American could find easy?

Comment author: Yvain 24 February 2012 01:32:59AM *  5 points [-]

Have you considered not taking anything? Most studies show multivitamins have somewhere between no effect and a slight negative effect except in groups with special nutritional issues. It might be possible to optimize further with the right combination (if the lack of effect from multivitamins is because they include both beneficial and harmful chemicals with a net zero effect) but I don't really think we know enough to do that right now.

Comment author: Alicorn 22 February 2012 06:59:58PM 9 points [-]

electronic charts have their built-in measures to prevent errors

This is at least a subset of the kind of software my dad works on. I remember him talking once about how someone was trying to get him to force doctors to dismiss every alert individually instead of just being able to select-all-dismiss, and my dad said that all that would do is waste time because the doctor would wind up mechanically dismissing 50 alerts without looking at them instead of doing it all at once, and trying to force the doctor to be conscientious with software was not the right way to go about making doctors be appropriately meticulous.

(Disclaimer: I probably misremember some substantial part of this.)

Comment author: Yvain 22 February 2012 07:08:22PM 9 points [-]

Yeah, the problem with introducing any safety procedure is that if you have to do it a thousand times a day, it eventually becomes mentally automated. It's like clicking "I agree" to EULAs, or "let this program access to the Internet" on one of those versions of Windows that would incessantly ask you whether you wanted to let a program access the Internet (never had one, but heard awful things from people who did).

I've never used a program like that, so I don't know whether the bar for alerts is set high enough that most alerts will be real errors, or whether it gets into so many nitpicky things (you're using an antihypertensive with another antihypertensive! What if that causes hypotension?!) that you eventually develop a reflex of clicking through them. I'd hope the former.

Comment author: Thomas 22 February 2012 06:25:59PM 2 points [-]

But I do think that once there are thousands of people and tens of millions of dollars in a field, you don't get to call it "hiding in plain sight" or "low hanging fruit" anymore.

How about the Heimlich method? There were millions of dollars and thousands of doctors in the field when he invented what he invented. Not a very high hanging fruit.

Comment author: Yvain 22 February 2012 06:54:23PM 4 points [-]
Comment author: Yvain 22 February 2012 06:48:45PM *  27 points [-]

Think about it: peptic and duodenal ulcer were fairly common, and so were effective antibiotics, starting in the mid-40s. . Every internist in the world – every surgeon – every GP was accidentally curing ulcers – not just one or twice, but again and again. For decades. Almost none of them noticed it, even though it was happening over and over, right in front of their eyes.

This is unfair. Modern attempts to eradicate h pylori use "triple therapy" of two different antibiotics plus a PPI, taken simultaneously, all for a unusually long period of time (one to two weeks). If you just give a patient a random antibiotic for some other disease, that's not going to produce astounding results: you need the PPI to alter the transport of the antibiotic and prevent it from getting immediately broken down in the acidic environment of the stomach, and you need both antibiotics in case the bacteria develop resistance to one. Although some doctors probably got lucky and eliminated h pylori by chance, it's not like every time anyone took an antibiotic it was curing their stomach ulcers and no one noticed.

Why else didn't people catch onto h pylori's role in stomach ulcers quicker? Well, at least 80% of people with h pylori don't have ulcers or any symptoms whatsoever, and 20% of people with ulcers don't have h pylori, so that's going to confuse people. Second, h pylori is hard to stain and very hard to culture, so all you have are these Bigfoot-esque rumors of "I saw this bacterium in the stomach...I think...no, I don't have any to show you." A bunch of studies looked for bacteria in the stomach and find none, because of previously mentioned staining and culturing problems. And whenever people tried treating ulcers with antibiotics - and they did do the studies - they get equivocal results because they weren't using the exact right drug combo to hit h pylori and keep it gone. The first person to successfully culture h pylori was the guy who won the Nobel Prize for discovering it.

Now, just from reading this article, you might believe that doctors are lax about looking at possible infective causes for chronic disease. Let's look at what PubMed has to say about possible infective causes of schizophrenia. There are 195 studies on "schizophrenia + influenza", 100 on "schizophrenia + herpesvirus", 84 on "schizophrenia + toxoplasma", 24 on "schizophrenia + cytomegalovirus", 17 on "schizophrenia + varicella", 10 on "schizophrenia + Lyme disease", 4 on "schizophrenia + neurocysticercosis", and one on "schizophrenia + trichinosis" (note that some of these studies are counted multiple times). This is not too atypical: after (among other things) the whole h pylori debacle people realized this was low hanging fruit and have been trying to pick it for the past thirty years.

I think that's generally a useful principle: if you know something is low-hanging fruit, then unless you're special so does everyone else, which means it's not low-hanging fruit anymore. And that's probably why h pylori seems so clear to Cochrane with thirty years of hindsight, while the people who figured it out at the time won the Nobel Prize, which generally isn't given for pointing out the obvious.

This illustrates an important point: doctors are lousy scientists, lousy researchers. They’re memorizers, not puzzle solvers.

Oh, be nice now.

Comment author: shminux 22 February 2012 04:05:11PM 6 points [-]

An obvious low-hanging fruit, hiding in plain sight is medical errors. Due to poor recognition and under-reporting, what we notice and read about is only the tail of the bell curve of severity.

Comment author: Yvain 22 February 2012 05:50:50PM *  21 points [-]

Interesting timing; I spent this morning in a risk management simulation: a mock hospital, with actors playing patients, surrounded by a bunch of doctors criticizing me when anything I did looked like it left potential open for a medical error. This was part of about fifty hours or so of training specifically in avoiding medical errors I've gone through in the past four years. Also, my record for "number of different doctors who have mentioned Atul Gawande to me in a single day" currently stands at three, and I'm expecting it to be broken before finishing school in May. It's safe to say any trend that's reached medical schools in southwestern Ireland is well established in the States, Britain, and the rest of the world.

There are dozens of journals, textbooks, and three-letter-acronym organizations dedicated solely to healthcare risk management, and most good hospitals will have their own on-site risk management team. Newer paper hospital charts have been specifically designed around risk management (for example, on the charts I used today, there are two spaces to sign for giving any drug: the signature of the person who gave it, and that of the person they checked it with to make sure it was safe), and those hospitals that have electronic charts have their built-in measures to prevent errors (for example, they'll automatically pop up an alert if you prescribe an unusual dose of any drug, or two drugs that interact with each other, or a drug a patient is known to be allergic to; sometimes only certain authoritative staff at the hospital have the password to override these alerts.)

Now, none of this means that risk management is any good. You're still making a bunch of chronically sleep-deprived and very busy people play a game of Chinese whispers with complicated biochemical data. And I sort of worry that some hospital manager has done the math and decided that the amount of money saved in malpractice suits is less than what it would cost to hire lots of extra doctors so each one is individually less busy and more awake.

But I do think that once there are thousands of people and tens of millions of dollars in a field, you don't get to call it "hiding in plain sight" or "low hanging fruit" anymore.

Comment author: gwern 20 February 2012 03:08:38AM *  8 points [-]

And of course to gwern, wedifrid, and several other people who knew much more about minerals than I did for not participating; that was helpful too.

Indeed. You can be sure that this contest has been entered into my personal mistakes file.

And I couldn't help but notice the resemblance of the lithium section to my own summary... :)

Comment author: Yvain 20 February 2012 11:28:01AM 6 points [-]

And I couldn't help but notice the resemblance of the lithium section to my own summary... :)

Kevin said the same thing, but I didn't read either of your writings until after the contest. I'm going to classify this one under "independent confirmation".

Comment author: Solvent 18 February 2012 01:27:55AM 3 points [-]

(you might be surprised how many medications we have no idea how they work)

How common are such medicines, in your experience?

Comment author: Yvain 18 February 2012 02:11:09AM *  32 points [-]

Literally no idea how they work? Not super common, although a paper on drug targets in Nature gives this list of drugs where they can't even begin to classify what sort of entity they act on.

But many other drugs are still incompletely understood, or only have very speculative mechanisms of actions. For example, we know that SSRI antidepressants increase levels of serotonin in the brain, but we're not really sure why that should treat depression (the hypothesis that depression is an imbalance of neurotransmitters was originally an ad hoc hypothesis explaining why these sorts of drugs seemed to treat it, although we've learned a lot since then).

Paracetamol (aka acetaminophen aka Tylenol) is actually pretty mysterious and still the subject of a lot of study, as are anaesthetics (which is too bad, because if we understood how anaesthetics worked, we'd have a promising lead in figuring out what consciousness is). Most psychiatric medications range somewhere from "incompletely understood" to "might as well be witchcraft", and lot of neurologic ones like some of the antiepileptics aren't much better.

I was surprised to learn recently that a lot of drug discovery is now being done by brute force. For example, they discovered new cystic fibrosis drug ivacaftor by dosing lung cells in solution with more or less every known organic chemical until one of them caused the chlorine concentration of the solution to change, which indicated that it had somehow solved the error in cellular chlorine transport that causes CF.

Comment author: Yvain 18 February 2012 01:33:44AM *  32 points [-]

A big thanks to Personalized Medicine for putting this on and subsidizing my learning about things I should have been motivated to learn anyway. Thanks also to User: TCB for proofreading my entry and catching several hilarious but unacceptable mistakes, which was very helpful. And of course to gwern, wedifrid, and several other people who knew much more about minerals than I did for not participating; that was helpful too.

Although I did the best I could with the time I had available and I stick to what I said, I absolutely agree with everyone who says the field is confused and I have relatively low confidence some of my recommendations. I agree with Zvi and Michael that there's plausible evidence that lowering sodium below 4g/day might not be beneficial and may even be harmful (though it's still better too low than too high), but there's also plausible evidence (some of which I cited) that lowering sodium is great and you should lower it as much as possible; I still think the balance of evidence is for the latter, but take the whole field with a grain of salt (pun intended). Calcium - I do stick to my position that it's more likely to harm than help in groups without elevated osteoporosis risk, but I could still totally be convinced of the opposite if there's some good study I missed or if I misinterpreted some statistic. As for lithium and iodine, I went into the research hoping I could avoid any weird recommendations that went against the general medical consensus, and clearly that plan didn't survive contact with the enemy; I feel a little awkward recommending them, but the whole "Bayesians can't privilege the null hypothesis" thing has forced my hand. Still, be really careful with those.

Also, Kevin's entry especially was really really good, and I recommend taking a look at it. It's very speculative, but it's interesting and well-founded speculation and I hope some of his theories get the research attention they deserve.

And I publicly precommit to donating at least half of the prize to charity, probably split between SIAI (got to remember to wait for their matching drive this time) and some warm fuzzy causes (is it weird that I think of malaria eradication as a warm fuzzy cause these days?)

Comment author: taw 18 February 2012 12:23:22AM 2 points [-]

To be honest I had much higher hopes for this competition than what I've seen.

Also seriously, top 2 entries make recommendations (lithium, paleo) with no plausible transmission mechanism. I know this is rampant is research, since you need to publish something, but I'd have hoped we'd knew better than that.

Comment author: Yvain 18 February 2012 12:55:27AM *  20 points [-]

By "transmission mechanism", do you mean "mechanism of action"? If so, when you say "with no plausible transmission mechanism", do you mean "whose mechanism of action has not yet been discovered" or "I believe it is implausible that a mechanism of action for this could exist"?

'Cause if you mean the former, the fact that we don't understand it shouldn't be a barrier to using it (you might be surprised how many medications we have no idea how they work), and if you mean the latter, well, as the old saying goes, "If it happens, it's possible".

This is doubly true for lithium, where even if you want to reject every single one of the studies I cited, we already know it works in higher dose as a treatment for bipolar disorder.

(and I hate to follow up an interesting rationality point with a boring discussion of pharmacology, but many of the effects of lithium are known or plausibly speculated upon, including protecting neurons against glutamate excitotoxicity; I am no neuroscientist, but I know excitotoxicity has a role in dementia, probably some psychiatric disorders, and in a bunch of common neurological causes of death.)

Comment author: Nectanebo 15 February 2012 09:08:31AM 2 points [-]

Isn't this statement also a clever argument for why you're not going to do it anyway, at least to an extent?

Comment author: Yvain 17 February 2012 01:38:44PM *  5 points [-]

Not a clever argument, more of an admission of current weakness. Admitting current weakness has the advantage of having the obvious next step of "consider becoming stronger".

But saying "Pursuing my interests would increase utility anyway" has the disadvantage of requiring no further actions. Which is fine if it's true, but if you evaluate the truth of the statement while you still have this potential source of bias lurking in the background, it might not be.

Comment author: cousin_it 14 February 2012 06:01:54PM *  26 points [-]

Does everyone else here think that putting aside your little quirky interests to do big important things is a good idea? It seems to me that people who choose that way typically don't end up doing much, even when they're strongly motivated, while people who follow their interests tend to become more awesome over time. Though I know Anna is going to frown on me for advocating this path...

Comment author: Yvain 15 February 2012 01:50:11AM *  11 points [-]

I think the flowchart for thinking about this question should look something like:

  1. If in a least convenient possible world where following your interests did not maximize utility, are you pretty sure you really would forego your personal interests to maximize utility? If no, go to 2; if yes, go to 3.

  2. Why are you even thinking about this question? Are you just trying to come up with a clever argument for something you're going to do anyway?

  3. Okay, now you can think about this question.

I can't answer your question because I've never gotten past 2.

Comment author: [deleted] 14 February 2012 07:30:20PM 4 points [-]

Link is dead. Would very much like to see web-comic. :)

In response to comment by [deleted] on No, Really, I've Deceived Myself
Comment author: Yvain 14 February 2012 09:22:58PM 16 points [-]
Comment author: CronoDAS 30 January 2012 01:44:04AM *  4 points [-]

Actually donating is slightly more risky than you make out here; general anesthesia is used in 75% of donations that involve actual bone marrow (and not peripheral blood stem cells), and in general, there are plenty of reasons to avoid getting surgery that involves anesthesia.

Comment author: Yvain 30 January 2012 11:47:17AM 1 point [-]

POCD lasting more than a few weeks is vanishingly uncommon in the young, and the overall mortality rate for anaesthesia is impressively low. The doctor who claimed you were more likely to die in a car crash driving to the hospital than from anaesthesia was exaggerating a little* - but if it's a very long drive to the hospital, say five hours each way, his argument might still stand a chance.

*estimate based only on deaths from anaesthesia. Usually if you're under anaesthesia, lots of other dangerous things are going on at the same time, but deaths from surgical trauma or the condition that necessitated surgery in the first place aren't counted in those numbers.

Comment author: Yvain 28 January 2012 04:20:12PM *  3 points [-]

PhronEasy, with the tagline "Making practical wisdom simple". No, wait, that's terrible.

My brother does some work for a professional naming company. Their prices are high, but they sound very professional (some of the stuff he talks about is a lot like what was mentioned here. If everything else fails, do be aware that such people exist.

Comment author: Yvain 28 January 2012 04:22:07PM 6 points [-]

Also, how about "Cogito"? Sounds related to thinking, but otherwise vague and non-threatening.

Comment author: Yvain 28 January 2012 04:20:12PM *  3 points [-]

PhronEasy, with the tagline "Making practical wisdom simple". No, wait, that's terrible.

My brother does some work for a professional naming company. Their prices are high, but they sound very professional (some of the stuff he talks about is a lot like what was mentioned here. If everything else fails, do be aware that such people exist.

Comment author: Yvain 26 January 2012 03:37:25PM *  38 points [-]

Good post, and upvoted, but I would phrase this part differently:

If a satisfactory answer to a hard question is not found quickly, System 1 will find a related question that is easier and will answer it.

The problem with a question like "What jobs make the most money" isn't so much that it's hard as that it's vague (or if you want to be harsh, "meaningless"). The question "How much would you contribute to save an endangered species" is even worse - if I were to actually answer it (by, for example, saying "Exactly two hundred seven dollars!"), you would be terribly confused and have no idea what I meant.

Seems to be a social norm that anyone asking an interlocutor to clarify a question is nitpicking and annoying, even though the overwhelming majority of questions people debate are meaningless as asked. People get rejected as poor conversational partners if they ask "To save all individuals in the species, or just to ensure at least one breeding pair...and are we talking per year, or pretending we have a 100% chance of saving them forever?", whereas if they say "We should pay whatever it takes!" they will be considered interesting even though that answer is clearly insane. It's no wonder that most people avoid becoming rationalists in such a situation.

Comment author: multifoliaterose 25 January 2012 05:26:10AM 5 points [-]

The only problem here is charity: I do think it may be morally important to be ambitious in helping others, which might even include taking a lucrative career in order to give money to charity. This is especially true if the Singularity memeplex is right and we're living in a desperate time that calls for a desperate effort. See for example Giving What You Can's powerpoint on ethical careers. At some point you need to balance how much good you want to do, with how likely you are to succeed in a career, with how miserable you want to make yourself - and at the very least rationality can help clarify that decision.

I don't know a single example of somebody who chose a career substantially less enjoyable than what they would otherwise have been doing in order to help people and successfully stuck to it. Do you?

Comment author: Yvain 25 January 2012 07:18:59PM *  4 points [-]

I don't know a single example of somebody who chose a career substantially less enjoyable than what they would otherwise have been doing in order to help people and successfully stuck to it. Do you?

I don't know a single example of somebody who chose a career substantially less enjoyable than what they would otherwise have been doing in order to help people in an efficient utilitarian way, full stop. I know juliawise was considering it, but I don't know what happened.

Do you know of anyone who tried and quit?

Comment author: Yvain 24 January 2012 06:03:05PM *  23 points [-]

This is also a repeat of the lesson that "you can often out-pace most philosophers simply by reading what today's leading scientists have to say about a given topic instead of reading what philosophers say about it."

On the other hand, rationality can be faster than science. And I'm feeling pretty good about positing three different forms of motivation, divided between model-free tendencies based on conditioning, and model-based goals, then saying we could use transhumanism to focus on the higher-level rational ones, without having read the particular neuroscience you're citing...

...actually, wait. I read as much of the linked paper as I could (Google Books hides quite a few pages) and I didn't really see any strong neuroscientific evidence. It looked like they were inferring the existence of the three systems from psychology and human behavior, and then throwing in a bit of neuroscience by mentioning some standard results like the cells that represent error in reinforcement learning. What I didn't see was a description of how three separate systems naturally fall out of brain studies. But I missed a lot of the paper - is there anything like that in there?

Comment author: [deleted] 23 January 2012 08:11:25PM 29 points [-]

Or, to put it more simply:

  1. Figure out what's going on, and what actions maximize returns, and do them.
  2. Do the thingy that worked before again!
  3. Avoid the unpleasant thing and go to the pleasant thing. Repeat as necessary.
Comment author: Yvain 24 January 2012 04:02:27AM *  2 points [-]

Is 2 operant/Skinnerian conditioning, and 3 classical/Pavolvian conditioning?

Comment author: Dreaded_Anomaly 23 January 2012 01:29:54AM 8 points [-]

Generic "ambition" is a serious case of putting the horse before the cart.

...cart before the horse?

Comment author: Yvain 23 January 2012 02:11:33AM *  7 points [-]

Every time I have ever used that phrase I have gotten it wrong, even when I specifically think about it beforehand and resolve not to get it wrong this time. I think it's because there are two related sayings, "keep the horse before the cart" and "don't put the cart before the horse", and I always sort of combine them.

Thank you.

Comment author: Yvain 22 January 2012 06:30:59AM *  10 points [-]

Solvent has just recorded a spectacular version of Hobbes' song

Comment author: Yvain 22 January 2012 04:51:00AM *  48 points [-]

Generic "ambition" is a serious case of putting the cart before the horse. If you have ambition to do something, that's great; if not, deciding you should be "ambitious" and then trying to figure out what to be ambitious about rarely ends well.

I think this is why I get creeped out by ambitious people sometimes. I'd much rather my doctor be passionate about medicine than be someone who decided medicine was more "prestigious" than nursing. As a personal anecdote, I am currently in medicine because I want to specialize in psychiatry. I am passionate about psychiatry and plan to be an awesome psychiatrist. I am not quite as passionate about organic medicine with its heart attacks and kidney infections, and although I work hard at it and am pretty good, some of my classmates who get up every morning super excited because they've dreamed of treating kidney infections their whole lives are better. I don't begrudge them this and if I ever got a kidney infection I'm going straight to them and not to the doctor who went into medicine as a subgoal of something else; if they ever get depression I hope they'll come to me for the same reason.

I understand it's the same in many other fields. Paul Graham writes that successful startup founders start with a problem they want to solve, eg Larry Page and Sergey Brin were frustrated with terrible online search; unsuccessful startup founders decide they would really like to earn fantastic amounts of money and only worry about what business they'll do it in as an afterthought.

The only problem here is charity: I do think it may be morally important to be ambitious in helping others, which might even include taking a lucrative career in order to give money to charity. This is especially true if the Singularity memeplex is right and we're living in a desperate time that calls for a desperate effort. See for example Giving What You Can's powerpoint on ethical careers. At some point you need to balance how much good you want to do, with how likely you are to succeed in a career, with how miserable you want to make yourself - and at the very least rationality can help clarify that decision.

Comment author: Yvain 22 January 2012 03:01:32AM *  4 points [-]

Yes. Suppose you had a gun with one trillion chambers. 1 is empty, 2 is empty, 3 contains a "negotiable" bullet, and 4 through one trillion all have non-negotiable bullets. How much would you pay to remove the bullet from 3?

Your first reaction is that forget it, it doesn't matter, you're quite certainly going to die anyway. Your second reaction is eh, what the heck, spend all your money on it, you're going to die anyway and you can't take it with you.

The only thing preventing you from spending all your money on bullet 3 is the infinitesimal chance that the gun will land on chamber 1 or 2, and then you'll regret having to live the rest of your life in poverty when you would have survived in any case. So although you don't really care because you're overwhelmingly likely to die either way, you balance your tiny chance of landing on bullet 3 and regretting you didn't buy it off against your tiny chance of landing on bullets 1 or 2 and regretting that you did.

This calculation remains the same whether we decrease the non-negotiable chambers to 0 or increase them to 3^^^3.

Comment author: bbarth 19 January 2012 09:28:05PM 4 points [-]

It's not a question of SIAI not being good enough for Yvain, it's a question of whether they might both do even better if he pursues something else. It clearly sounds like he's pursing a different path than joining SIAI now, so he must have done at least some of the math. He's in med school according to his webpage, so I suspect his prospects for helping the cause might be higher if he does well as a doctor and sends every dime he doesn't need (say his salary as a doctor less $36k/yr) to SIAI. It certainly seems like it might be a waste of his current efforts to drop his medical aspirations and become a curriculum producer at SIAI, but I might be suffering from a form of the Sunk Cost Fallacy here.

Comment author: Yvain 19 January 2012 09:46:52PM *  8 points [-]

Thanks to the magic of guilds, all new trainee doctor jobs in the US start on July 1st*. If I don't get a job by then, I will probably have to wait until next July and find something to occupy me and provide me with money for a year. Hence my comment that I would be interested if my job search fell flat.

Even though it doesn't look like it sometimes, I do give at least five minutes thought to most of my major life decisions.

*which is why some people have very reasonably argued that you should avoid hospitals at that time of year.

Comment author: bbarth 19 January 2012 03:38:37PM *  5 points [-]

I guess my points were a little too obtuse. People with even a handful of these skills get paid a lot more than you're offering (e.g. school teachers have curriculum design and teaching experience, and generally make much more than $36k/yr). Clearly, stating the salary is "upfront" about the salary, but that wasn't my complaint. My complaint was that it appears that by offering a well below market salary you're looking for a fellow traveler/true believer/movement participant who is so highly dedicated to the cause that they are willing to sacrifice a good chunk of their potential earnings to advance SIAI's goals. If that's the case, then you should state it directly. If it's not the case, then another possibility that comes to mind is that you're hoping to exploit the passion of a young person who feels strongly about the cause but doesn't realize what they're worth on the open market.

My concern is that by not stating anything about this obviously (to me) below market salary, you're leaving your motivations open to serious question. I think it better to lay out some sort of reasoning behind it than to leave it ambiguous.

Comment author: Yvain 19 January 2012 08:57:26PM 9 points [-]

They're offering 150% of the average US income during a recession with 9% unemployment as starting salary for a potentially entry-level position doing satisfying creative work for an organization that could actually improve the world. I like money as much as anyone else, and I would fight for this job if I weren't otherwise engaged. If my hunt for residency positions this summer falls flat, I might still try to fight for it.

Comment author: Yvain 19 January 2012 08:55:12PM *  32 points [-]

Your link doesn't work because you used http;/ instead of http://

(I noticed that in less than a minute. May I have $100?)

Comment author: thomblake 19 January 2012 02:39:09PM 3 points [-]

For $3k a month, you're practically looking for volunteers.

Comment author: Yvain 19 January 2012 08:48:30PM 2 points [-]

They're offering 150% of the average US income during a recession with 9% unemployment as starting salary for an entry-level position doing satisfying creative work for an organization that could actually improve the world. I like money as much as anyone else, and I would fight for this job if I weren't otherwise engaged. If my hunt for residency positions this summer falls flat, I might still try to fight for it.

Comment author: Solvent 06 January 2012 05:48:54AM 3 points [-]

I am working on my entry to this competition right now. I am having great fun.

How many other people are working on entries?

What happens if, for example, me and three other people submit entries, but mine is mediocre? Do I still get a prize?

Comment author: Yvain 18 January 2012 02:09:31AM *  4 points [-]

I surprised myself by managing to submit an entry to the contest an hour or two before deadline.

I don't want to say it wasn't very good, but...well, I could either discuss all the minerals, discuss a mineral or two in suitable depth, or vastly exceed the recommended word count and the amount of time I could reasonably devote to this project. I won't say what I chose, since that might bias peer review, but looking at the peer reviews, it looks like many of the entries had very different interpretations on where to go with the question and a lot of it is going to be who was lucky enough to interpret it in the same way the judges do.

The ID numbers of the entries I was told to peer review are kind of obtuse. I guess I won't speculate on what they might mean about the number of entries publicly, lest giving secret peer-review information be against the rules or something. But I am very curious how many people entered. I told a friend to enter the contest with a single-sentence entry saying just "Minerals are good for you and you should eat more of them"; just in case the contest had fewer than five entries it would be the easiest $500 she ever earned. She quite properly refused.

Also, looking over my desktop today I realized there's like a 25% chance I accidentally turned in a super-rough-draft with a similar name to my final copy. So, um, if anyone got a paper for peer review where the references are things like [small-calcium-study] or [that-one-experiment-with-the-potassium] and there's no abstract or recommendations, let me know now so I can disqualify myself and avoid further humiliation.

Comment author: Yvain 17 January 2012 09:31:14PM 18 points [-]

I have the same experience, although it started long before I started reading Less Wrong. And it's not limited to skepticism; it also strikes when people are expressing what I consider very wrong political or sometimes even artistic views

It has never stricken me as disliking people before; there are people with views I find ridiculous whose company I can enjoy so long as they are not expressing those views at the moment. And it would not bother me if they were just to assert "I'm a fundamentalist / a fascist / whatever". They would have to be making arguments for their position.

I do not have a good explanation either, but perhaps I view it as a sort of attack. If fundamentalism is true, then atheism is not true, and I am stupid or at least a very bad truth-seeker for being an atheist. Letting yourself be attacked, even indirectly, without defending yourself is hard.

Comment author: Yvain 16 January 2012 08:44:56AM 0 points [-]

Is anyone available to give me a ride from Cincinnati city center to the meetup? I can pay in gas money or interesting conversation.

Comment author: TheOtherDave 13 January 2012 06:20:39PM 3 points [-]

If I'm understanding you right, you are assuming that a ratio not skewed to favor males or females would result in no more unmarried males than a ratio skewed to favor females.

Am I understanding you right?

If so... that seems unlikely to me. Can you say more about why you expect it?

Comment author: Yvain 13 January 2012 09:38:20PM 5 points [-]

You are of course right, although I stick to the general point that we have to distinguish an effect of fewer unmarried males from an effect that does not directly involve fewer unmarried males.

Comment author: Viliam_Bur 13 January 2012 10:14:58AM 5 points [-]

It's inspiring to know that we really can create a better and more peaceful society, just by pursuing some simple ideals like killing fifty percent of males.

I think some famous feminist recommended unspecified disappearing of 90% of males to make the world a better place, but right now I can't find the quote.

However, from scientific point of view, this situation could be an inspiration for some interesting experiments. If you remove dominant males from one generation, how long does it take until the next generation creates new ones? (I would expect one or two at most.)

Comment author: Yvain 13 January 2012 06:13:21PM 14 points [-]

Well, there's some evidence that having a ratio skewed in favor of males in a society increases violence. I don't know if you could make the contrary claim that one skewed in favor of females would actually decrease violence.

You'd have to distinguish between the relatively uncontroversial claim that unmarried males (who'll be more common with a pro-male sex ratio) are the most likely group to commit violence, versus the very speculative claim that even if all males have sufficient opportunity to marry off, more female presence will make them less violent - either because "female values" dominate the society, or because the less competition for "sufficiently good" mates they expect, the less competitive they will act.

Comment author: Yvain 12 January 2012 08:54:00PM 41 points [-]

It's inspiring to know that we really can create a better and more peaceful society, just by pursuing some simple ideals like killing fifty percent of males.

(I didn't fully understand that part. So the males who ate the infected meat didn't spread the TB to females? And when the male/female ratio changed, that shifted the social dynamics and made everyone more peaceful because there was less reason for status competition? Or because the next generation had only nonviolent female role models and so learned less violence?)

Comment author: AspiringKnitter 08 January 2012 10:57:04PM 1 point [-]

Second, that you find mystical experiences by other people inherently hard to believe but you believe your own because you are a normal sane person (1,2,5).

Unless I know them already. Once I already know people for honest, normal, sane people ("normal" isn't actually required and I object to the typicalist language), their miracle stories have the same weight as my own. Also, miracles of more empirically-verifiable sorts are believable when vetted by snopes.com.

If we're going to go gender-neutral, I recommend "eir", just because I think it's the most common gender neutral pronoun on this site and there are advantages to standardizing this sort of thing.

Xe is poetic and awesome. I'm hoping it'll become standard English. To that end, I use it often.

(including changing as a person)

I read your first link and I'm very surprised because I didn't expect something like that. It would be interesting to talk to that person about this.

So either only a tiny fraction of a percent of people are open to the knowledge - so tiny that you could not reasonably expect yourself to be among them -

Is that surprising? First of all, I know that I already converted to Christianity, rather than just having assumed it always, so I'm already more likely to be open to new facts. And second, I thought it was common knowledge around these parts that most people are really, really bad at finding the truth. How many people know Bayes? How many know what confirmation bias is? Anchoring? The Litany of Tarski? Don't people on this site rail against how low the sanity waterline is? I mean, you don't disagree that I'm more rational than most Christians and Muslims, right?

Different studies show somewhere from a third to half of Americans having mystical experiences, including about a third of non-religious people who have less incentive to lie. Five percent of people experience them "regularly".

Do they do this by using tricks like Multiheaded described? Or by using mystical plants or meditation? (I know there are Christians who think repeating a certain prayer as a mantra and meditating on it for a long time is supposed to work... and isn't there, or wasn't there, some Islamic sect where people try to find God by spinning around?) If so, that really doesn't count. Is there another study where that question was asked? Because if you're asserting that mystical experiences can be artificially induced by such means in most if not all people, then we're in agreement.

Well, okay, but this seems to be an argument from force, sort of "Jehovah is a god and Astarte a demon because if I say anything else, Jehovah will torture me". It seems to have the same form as "Stalin is not a tyrant, because if I call Stalin a tyrant, he will kill me, and I don't want that!"

I was thinking more along the lines of "going to hell is a natural consequence of worshiping Astarte", analogous to "if I listen to my peers and smoke pot, I won't be able to sing, whereas if I listen to my mother and drink lots of water, I will; therefore, my mother is right and listening to my peers is bad". I hadn't even considered it from that point of view before.

Comment author: Yvain 10 January 2012 04:54:35AM *  7 points [-]

Is that surprising? ... Don't people on this site rail against how low the sanity waterline is? I mean, you don't disagree that I'm more rational than most Christians and Muslims, right?

No, I suppose it's not surprising. I guess I misread the connotations of your claim. Although I am still not certain I agree: I know some very rational and intelligent Christians, and some very rational and intelligent atheists (I don't really know many Muslims, so I can't say anything about them). At some point I guess this statement is true by definition, since we can define open-minded as "open-minded enough to convert religion if you have good enough evidence to do so." But I can't remember where we were going with this one so I'll shut up about it.

Do they do this by using tricks like Multiheaded described? Or by using mystical plants or meditation? (I know there are Christians who think repeating a certain prayer as a mantra and meditating on it for a long time is supposed to work... and isn't there, or wasn't there, some Islamic sect where people try to find God by spinning around?) If so, that really doesn't count. Is there another study where that question was asked? Because if you're asserting that mystical experiences can be artificially induced by such means in most if not all people, then we're in agreement.

I was unable to find numerical data on this. I did find some assertions in the surveys that some of the mystical experience was untriggered, I found one study comparing 31 people with triggered mystical experience to 31 people with untriggered mystical experience (suggesting it's not too hard to get a sample of the latter), and I have heard anecdotes from people I know about having untriggered mystical experience.

Honestly I had never really thought of that as an important difference. Keep in mind that it's really weird that the brain responds to relatively normal stressors, like fasting or twirling or staying still for two long, by producing this incredible feeling of union with God. Think of how surprising this would be if you weren't previously aware of it, how complex a behavior this is, as opposed to something simpler like falling unconscious. The brain seems to have this built-in, surprising tendency to have mystical experiences, which can be triggered by a lot of different things.

As someone in the field of medicine, this calls to mind the case of seizures, another unusual mental event which can be triggered in similar conditions. Doctors have this concept called the "seizure threshold". Some people have low seizure thresholds, other people high seizure thresholds. Various events - taking certain drugs, getting certain diseases, being very stressed, even seeing flashing lights in certain patterns - increases your chance of having a seizure, until it passes your personal seizure threshold and you have one. And then there are some people - your epileptics - who can just have seizures seemingly out of nowhere in the course of everyday life (another example is that some lucky people can induce orgasm at will, whereas most of us only achieve orgasm after certain triggers).

I see mystical experiences as working a lot like seizures - anyone can have one if they experience enough triggers, and some people experience them without any triggers at all. It wouldn't be at all parsimonous to say that some people have this reaction when they skip a few meals, or stay in the dark, or sit very still, and other people have this reaction when they haven't done any of these things, but these are caused by two completely different processes.

I mean, if we already know that dreaming up mystical experiences is the sort of thing the brain does in some conditions, it's a lot easier to expand that to "and it also does that in other conditions" than to say "but if it happens in other conditions, it is proof of God and angels and demons and an entire structure of supernatural entities."

I was thinking more along the lines of "going to hell is a natural consequence of worshiping Astarte", analogous to "if I listen to my peers and smoke pot, I won't be able to sing, whereas if I listen to my mother and drink lots of water, I will; therefore, my mother is right and listening to my peers is bad". I hadn't even considered it from that point of view before.

The (relatively sparse) Biblical evidence suggests an active role of God in creating Hell and damning people to it. For example:

"This is how it will be at the end of the age. The angels will come and separate the wicked from the righteous and throw them into the blazing furnace, where there will be weeping and gnashing of teeth." (Matthew 13:49)

"Depart from me, you accursed, into the eternal fire that has been prepared for the devil and his angels!" (Matthew 25:41)

"If anyone’s name was not found written in the book of life, that person was thrown into the lake of fire." (Revelations 20:15)

"God did not spare angels when they sinned, but sent them to hell, putting them into gloomy dungeons to be held for judgment" (2 Peter 2:4)

"Fear him who, after the killing of the body, has power to throw you into hell. Yes, I tell you, fear him." (Luke 12:5)

That last one is particularly, um, pleasant. And it's part of why it is difficult for me to see a moral superiority of Jehovah over Astarte: of the one who's torturing people eternally, over the one who fails to inform you that her rival is torturing people eternally.

Comment author: AspiringKnitter 07 January 2012 08:51:02PM 2 points [-]

Since you use the names Jehovah and Astarte, I'll follow suit, though they're not the names I prefer.

The difference would be that if worship of Jehovah gets you eternal life in heaven, and worship of Astarte gets you eternal torture and damnation, then you should worship Jehovah and not Astarte. Also, if Astarte knows this, but pretends otherwise, then Astarte's a liar.

If you already believe that there are multiple factions of supernatural beings, some of whom push true religions and others of whom push false ones, then noticing that some supernatural being is trying to push you toward Christianity provides zero extra evidence that Christianity is true.

Not quite. I only believe in "multiple factions of supernatural beings" (actually only two) because it's implied by Christianity being true. It's not a prior belief. If Christianity is false, one or two or fifteen or zero omnipotent or slightly-powerful or once-human or monstrous gods could exist, but if Christianity is false I'd default to atheism, since if my evidence for Christianity proved false (say, I hallucinated it all because of some undiagnosed mental illness that doesn't resemble any currently-known mental illness and only causes that one symptom) without my gaining additional evidence for some other religion or non-atheist cosmology, I'd have no evidence for anything spiritual. Or do I misunderstand? I'm confused.

Why should you trust the supernatural beings who have taken an interest in your case, as opposed to the supernatural beings apparently from a different faction who caused the seemingly miraculous revelations in this person and this person's lives?

Being, singular, first of all.

  1. I already know myself, what kind of a person I am. I know how rational I am. I know how non-crazy I am. I know exactly the extent to which I've considered illness affecting my thoughts as a possible explanation.

  2. I know I'm not lying.

  3. The first person became an apostate, something I've never done, and is still confused years later. The second person records only the initial conversion, while I know how it's played out in my own life for several years.

  4. The second person is irrationally turned off by even the mere appearance of Catholicism and Christianity in general because of terrible experiences with Catholics.

  5. I discount all miracle stories from people I don't know, including Christian and Jewish miracle stories, which could at least plausibly be true. I discount them ALL when I don't know the person. In fact, that means MOST of the stories I hear and consider unlikely (without passing judgment when I have so little info) are stories that, if true, essentially imply Christianity, while others would provide evidence for it.

  6. And knowing how my life has gone, I know how I've changed as a person since accepting Jesus, or Jehovah if that's the word you prefer. They don't mention drastic changes to their whole personalities to the point of near-unrecognizability even to themselves. In brief: I was unbelievably awful. I was cruel, hateful, spiteful, vengeful and not a nice person. I was actively hurtful toward everyone, including immediate family. After finding Jesus, I slowly became a less horrible person, until I got to where I am now. Self-evaluation may be somewhat unreliable, but I think the lack of any physical violence recently is a good sign. Also, rather than escalating arguments as far as possible, when I realize I've lashed out, I deliberately make an effort not to fall prey to consistency bias and defend my actions, but to stop and apologize and calm down. That's something I would not have done-- would not have WANTED to do, would not have thought was a good idea, before.

And you agree that Astarte performs minor miracles of the same caliber all the time to try to convince her own worshippers.

I don't know (I only guess) what Astarte does to xyr worshipers. I'm conjecturing; I've never prayed to xem, nor have I ever been a Wiccan or any other type of non-Christian religion. But I think I ADBOC this statement; if said by me, it would have sounded more like "Satan makes xyrself look very appealing".

(I'm used to a masculine form for this being. You're using a feminine form. Rather than argue, I've simply shifted my pronoun usage to an accurate-- possibly more accurate-- and less loaded set of pronouns.)

Also, my experience suggests that if something is good or evil, and you're open to the knowledge, you'll see through any lies or illusions with time. It might be a lot of time-- I'll confess I recently got suckered into something for, I think, a couple of years, when I really ought to have known better much sooner, and no, I don't want to talk about it-- but to miss it forever requires deluding yourself.

(Not, as we all know, that self-delusion is particularly rare...)

So all that your feelings indicate is that some supernatural being is trying to push you toward Christianity.

That someone is trying to convince me to be a Christian or that I perceive the nature of things using an extra sense.

Giving someone a weak feeling of serenity or repulsion is, as miracles go, not a very flashy one.

Strength varies. Around the time I got to the fourth Surah of the Koran, it was much flashier than anything I've seen since, including everything previously described (on the negative side) at incredible strength plus an olfactory hallucination. And the result of, I think, two days straight of Bible study and prayer at all times constantly... well, that was more than a weak feeling of serenity. But on its own it'd be pretty weak evidence, because I was only devoting so much time to prayer because my state of mind was so volatile and my thoughts and feelings were unreliable. It's only repetitions of that effect that let me conclude that it means what I've already listed, after controlling for other possibilities that are personal so I don't want to talk about it. Those are rare extremes, though; normally it's not as flashy as those.

you seem to think that Astarte's main testable claim - that she does magic for her followers - is true.

I consider it way likelier than you do, anyway. I'm only around fiftyish percent confidence here. But that's only one aspect of it. Their religion also claims to cause changes in its followers along the lines of "more in tune with the Divine" or something, right? So if there are any overlapping claims about morality, that would also be testable-- NOT absolute morality of the followers, but change in morality on mutually-believed-in traits, measuring before and after conversion, then a year on, then a few years on, then several years on. Of course, I'm not sure how you'll ever get the truth about how moral people are when they think no one's watching...

Comment author: Yvain 08 January 2012 05:36:17PM *  14 points [-]

Sorry - I used "Astarte" and the female pronoun because the Wiccans claim to worship a Goddess, and Astarte was the first female demon I could think of. If we're going to go gender-neutral, I recommend "eir", just because I think it's the most common gender neutral pronoun on this site and there are advantages to standardizing this sort of thing.

The difference would be that if worship of Jehovah gets you eternal life in heaven, and worship of Astarte gets you eternal torture and damnation, then you should worship Jehovah and not Astarte.

Well, okay, but this seems to be an argument from force, sort of "Jehovah is a god and Astarte a demon because if I say anything else, Jehovah will torture me". It seems to have the same form as "Stalin is not a tyrant, because if I call Stalin a tyrant, he will kill me, and I don't want that!"

Not quite. I only believe in "multiple factions of supernatural beings" (actually only two) because it's implied by Christianity being true.

It sounds like you're saying the causal history of your belief should affect the probability of it being true.

Suppose before you had any mystical experience, you had non-zero probabilities X of atheism, Y of Christianity (in which God promotes Christianity and demons promote non-Christian religions like Wicca), and Z of any non-Christian religion (in which God promotes that religion and demons promote Christianity).

Then you experience an event which you interpret as evidence for a supernatural being promoting Christianity. This should raise the probability of Y and Z an equal amount, since both theories seem to equally predict this would happen.

You could still end up a Christian if you started off with a higher probability Y than Z, but it sounds like you weren't especially interested in Christianity before your mystical experience, and the prior for Z is higher than Y since there are so many more non-Christian than Christian religions.

Being, singular, first of all...

I understand you as having two categories of objections: first, objections that the specific people in the Islamic conversion stories are untrustworthy or their stories uninteresting (3,4,6). Second, that you find mystical experiences by other people inherently hard to believe but you believe your own because you are a normal sane person (1,2,5).

The first category of objections apply only to those specific people's stories. That's fair enough since those were the ones I presented, but they were the ones I presented because they were the first few good ones I found in the vast vast vast vast VAST Islamic conversion story literature. I assume that if you were to list your criteria for believability, we could eventually find some Muslim who experienced a seemingly miraculous conversion who fit all of those criteria (including changing as a person) - if it's important to you to test this, we can try.

The second category of objections is more interesting. Different studies show somewhere from a third to half of Americans having mystical experiences, including about a third of non-religious people who have less incentive to lie. Five percent of people experience them "regularly". Even granted that some of these people are lying and other people categorize "I felt really good" as a mystical experience, I don't think denying that these occur is really an option.

The typical view that people need to be crazy, or on the brink of death, or uneducated, or something other than a normal middle class college-educated WASP adult in order to have mystical experiences also breaks down before the evidence. According to Greeley 1975 and Hay and Morisy 1976, well-educated upper class people are more likely to have mystical experiences, and Hay and Morisy 1978 found that people with mystical experiences are more likely to be mentally well-balanced.

Since these experiences occur with equal frequency among people of all religion and even atheists, I continue to think this supports either the "natural mental process" idea or the "different factions of demons" idea - you can probably guess which one I prefer :)

Also, my experience suggests that if something is good or evil, and you're open to the knowledge, you'll see through any lies or illusions with time.

There are 1.57 billion Muslims and 2.2 billion Christians in the world. Barring something very New-Agey going on, at least one of those groups believes an evil lie. The number of Muslims who convert to Christianity at some point in their lives, or vice versa, is only a tiny fraction of a percent. So either only a tiny fraction of a percent of people are open to the knowledge - so tiny that you could not reasonably expect yourself to be among them - or your experience has just been empirically disproven.

(PS: You're in a lot of conversations at once - let me know if you want me to drop this discussion, or postpone it for later)

Comment author: Alejandro1 07 January 2012 08:03:05AM 1 point [-]

Obviously I can't speak for AK, but maybe she believes that she has been epistemically lucky. Compare the religious case:

"I had this experience which gave me evidence for divinity X, so I am going to believe in X. Others have had analogous experiences for divinities Y and Z, but according to the X religion I adopted those are demonic, so Y and Z believers are wrong. I was lucky though, since if I had had a Y experience I would have become a Y believer".

with philosophical cases like the ones Alicorn discusses there:

"I accept philosophical position X because of compelling arguments I have been exposed to. Others have been exposed to seemingly compelling arguments for positions Y and Z, but according to X these arguments are flawed, so Y and Z believers are wrong. I was lucky though, since if I had gone to a university with Y teachers I would have become a Y believer".

It may be that the philosopher is also being irrational here and that she could strive more to trascend her education and assess X vs Y impartially, but in the end it is impossible to escape this kind of irrationality at all levels at once and assess beliefs from a perfect vaccuum. We all find some things compelling and not others because of the kind of people we are and the kind of lives we have lived, and the best we can get is reflective equilibrium. Recursive justification hitting bottom and all that.

The question is whether AK is already in reflective equilibrium or if she can still profit from some meta-examination and reassess this part of her belief system. (I believe that some religious believers have reflected enough about their beliefs and the counterarguments to them that they are in this kind of equilibrium and there is no further argument from an atheist that can rationally move them - though these are a minority and not representative of typical religious folks.)

Comment author: Yvain 08 January 2012 12:55:23AM 2 points [-]

See my response here - if Alicorn is saying she knows the other side has arguments exactly as convincing as those which led her to her side, but she is still justified to continue believing her side more likely than the other, I disagree with her.

Comment author: AspiringKnitter 07 January 2012 04:43:08AM *  15 points [-]

I should probably point out at this point that Wiccans (well, at least those whom I'd met), consider this point of view utterly misguided and incredibly offensive.

That's a bizarre thing to say. Is their offense evidence that I'm wrong? I don't think so; I'd expect it whether or not they worship demons. Or should I believe something falsely because the truth is offensive? That would go against my values-- and, dare I say it, the suggestion is offensive. ;) Or do you want me to lie so I'll sound less offensive? That risks harm to me (it's forbidden by the New Testament) and to them (if no one ever tells them the truth, they can't learn), as well as not being any fun.

No one likes to be called a "demon-worshiper",

What is true is already so, Owning up to it doesn't make it worse. Not being open about it doesn't make it go away.

especially when one is generally a nice person whose main tenet in life is a version of "do no harm".

Nice people like that deserve truth, not lies, especially when eternity is at stake.

flat-out calling a whole group of people "demon-worshipers" tends to inflame passions rather quickly,

So does calling people Cthulhu-worshipers. But when you read that article, you agreed that it was apt, right? Because you think it's true. You guys sure seem quick to tell me that my beliefs are offensive, but if I said the same to you, you'd understand why that's beside the point. If Wiccans worship demons, I desire to believe that Wiccans worship demons; if Wiccans don't worship demons, I desire to believe that Wiccans don't worship demons. Sure, it's offensive and un-PC. If you want me to stop believing it, tell me why you think it's wrong.

Comment author: Yvain 07 January 2012 07:23:08AM *  18 points [-]

I like your post (and totally agree with the first paragraph), but have some concerns that are a little different from Bugmaster's.

What's the exact difference between a god and a demon? Suppose Wicca is run by a supernatural being (let's call her Astarte) who asks her followers to follow commendable moral rules, grants their petitions when expressed in the ritualistic form of spells, and insists she will reward the righteous and punishes the wicked. You worship a different supernatural being who also asks His followers to follow commendable moral rules, grants their petitions when expressed in the ritualistic form of prayer, and insists He will reward the righteous and punish the wicked. If both Jehovah and Astarte exist and act similarly, why name one "a god" and the other "a demon"? Really, the only asymmetry seems to be that Jehovah tries to inflict eternal torture on people who prefer Astarte, where Astarte has made no such threats among people who prefer Jehovah, which is honestly advantage Astarte. So why not just say "Of all the supernatural beings out there, some people prefer this one and other people prefer that one"?

I mean, one obvious answer is certainly to list the ways Jehovah is superior to Astarte - the one created the Universe, the other merely lives in it; the one is all-powerful, the other merely has some magic; the one is wise and compassionate, the other evil and twisted. But all of these are Jehovah's assertions. One imagines Astarte makes different assertions to her followers. The question is whose claims to believe.

Jehovah has a record of making claims which seem to contradict the evidence from other sources - the seven-day creation story, for example. And He has a history of doing things which, when assessed independently of their divine origin, we would consider immoral - the Massacre of the Firstborn in Exodus, or sanctioning the rape, enslavement, infanticide, and genocide of the Canaanites. So it doesn't seem obvious at all that we should trust His word over Astarte's, especially since you seem to think that Astarte's main testable claim - that she does magic for her followers - is true.

Now, you've already said that you believe in Christianity because of direct personal revelation - a sense of serenity and rightness when you hear its doctrines, and a sense of repulsion from competing religions, and that this worked even when you didn't know what religion you were encountering and so could not bias the result. I upvoted you when you first posted this because I agree that such feelings could provide some support for religious belief. But that was before you said you believed in competing supernatural beings. Surely you realize how difficult a situation that puts you in?

Giving someone a weak feeling of serenity or repulsion is, as miracles go, not a very flashy one. One imagines it would take only simple magic, and should be well within the repertoire of even a minor demon or spirit. And you agree that Astarte performs minor miracles of the same caliber all the time to try to convince her own worshippers. So all that your feelings indicate is that some supernatural being is trying to push you toward Christianity. If you already believe that there are multiple factions of supernatural beings, some of whom push true religions and others of whom push false ones, then noticing that some supernatural being is trying to push you toward Christianity provides zero extra evidence that Christianity is true.

Why should you trust the supernatural beings who have taken an interest in your case, as opposed to the supernatural beings apparently from a different faction who caused the seemingly miraculous revelations in this person and this person's lives?

Comment author: [deleted] 06 January 2012 10:47:06PM 14 points [-]

You're doing that thing where you write like Yudkowsky again. It's kind of hot.

In response to comment by [deleted] on What Curiosity Looks Like
Comment author: Yvain 07 January 2012 03:09:05AM 36 points [-]

And yet the conclusion is so...Hansonesque.

Every week I talk to people who say they are trying to figure out the truth about something. When I ask them a few questions about it, I often learn that they know almost nothing of logic, probability theory, argument, scientific method, epistemology, artificial intelligence, human cognitive science, or debiasing techniques...I conclude that they probably want to feel they are truth-seeking, or they want to signal a desire for truth-seeking, or they might even self-deceivingly "believe" that they place a high value on knowing the truth. But their actions show that they aren't trying very hard to have true beliefs.

Really? What percent of people are aware of the existence of cognitive biases? One percent? At least I wouldn't expect more than that to realize that probability theory or artificial intelligence bear upon questions in seemingly unrelated fields like philosophy or medicine.

And of people who know of the existence of cognitive biases, how many are even capable of genuinely entertaining the thought that they themselves might be biased, as opposed to Rush Limbaugh or unethical pharmaceutical researchers or all those silly people who disagree with them?

And of people who are worried about cognitive biases, how many have access to "debiasing techniques"? I'm not going to put a percent on this one because it's pretty vague, but outside of Less Wrong and a few ahead-of-the-game finance companies, you can't exactly go on Amazon and buy Debiasing for Dummies.

I think I agree with the conclusion (well, maybe, since I don't know enough psychodynamics to really be able to cash out a phrase like "their actions show that they aren't trying very hard) but this particular argument breaks Hanlon's Razor aka the Generalized Anti-Hanson Principle.

Comment author: Yvain 05 January 2012 02:28:43AM *  25 points [-]

You suggest "professional subjects" would try to decide whether they're in the placebo or treatment arm. But this seems difficult (people who are badly-off enough to consider "professional patient" a career choice probably are going to have a hard time testing for chemical compositions, and as you mention placebos can cause placebo side effects), unnecessary (researchers can never say "You didn't give us the effect we expected, so we think you're cheating and we're not going to pay you for this study and we're blacklisting you for future studies" without opening a can of worms), and contradictory to the observed results (if fake patients did this successfully, it would mean a greater difference between placebo and experimental, since placebo would report no change and experimental would report strong positive effects, but the conundrum to be explained is that antidepressant studies seem to be showing less efficacy over time.)

I think it's much more plausible that if these "professional subjects" existed, they would just say "Yes, the drug worked great" on the theory that this is what the researchers want to hear (they may not understand the idea of placebo-control, and even if they do, they know that it means they can totally get away with being "cured" by a fake pill). This would present as placebos appearing more effective than they did in previous, more honest trials - which seems to be happening - and a decrease in positive findings.

I would be surprised if this were large enough to make a big difference, but it's certainly worth considering. I wonder if there are any meta-analyses comparing studies recruiting by public advertising, versus studies recruiting by a doctor or institution referring their own patient population.

Comment author: fortyeridania 02 January 2012 11:49:23AM 6 points [-]

Is it more accurate to put it thus because Yvain-2012 disagrees with Yvain-2004 on this issue?

Comment author: Yvain 03 January 2012 02:12:29AM *  29 points [-]

I don't know if there's enough of a specific, meaningful claim there for me to disagree with, but Yvain-2012 probably would not have written those same words. Yvain-2012 would probably say he sometimes feels creeped out by the levels of signaling that go on in the skeptical community and thinks they sometimes snowball into the ridiculous, but that the result is prosocial and they are still performing a service.

(really I can only speak for Yvain-2011 at this point; my acquaintance with Yvain-2012 has been extremely brief)

Comment author: Kutta 01 January 2012 11:23:25AM *  9 points [-]

Most people you know are probably weak skeptics, and I would probably fit this definition in several ways. "Strong skeptics" are the people who write The Skeptics' Encyclopedia, join the California Skeptics' League, buy the Complete Works of James Randi, and introduce themselves at parties saying "Hi, I'm Ted, and I'm a skeptic!". Of weak skeptics I approve entirely. But strong skeptics confused me for a long while. You don't believe something exists. That seems like a pretty good reason not to be too concerned with it.

Edit: authorial instance specified on popular demand.

Comment author: Yvain 02 January 2012 05:40:15AM *  12 points [-]

More accurately, Yvain-2004

Comment author: dlthomas 26 December 2011 06:18:32AM *  2 points [-]
  1. Personal observation.
  2. Fisher's denial that smoking contributed to lung cancer.
Comment author: Yvain 26 December 2011 07:56:33AM 0 points [-]

I was mostly going to say (1), but (2) certainly crossed my mind as an example of the other sort of error.

Comment author: Yvain 25 December 2011 09:59:02PM *  21 points [-]

I do worry sometimes that the pendulum has swung too far in the other direction, and that people are starting using correlation-causation as an I'm-smarter-than-you sort of status signal - that is, once people pass a certain intelligence level I worry less about them claiming Facebook causes the Greek debt crisis because they're correlated, and more about them hearing a very well-conducted study showing an r = .98 correlation between some disease and some risk factor, and instead of agreeing we should investigate further they just say "HA! GOTCHA! CORRELATION'S NOT THE SAME THING AS CAUSATION!"

I mean, I admit it's an important lesson, as long as people remember it's just a caution against being too certain of a causal relationship, and not a guarantee that a correlation provides absolutely no evidence.

Comment author: Yvain 24 December 2011 04:13:44AM 4 points [-]

We can hack this a little by working while lying down, though many people have trouble focusing given the implied lack of focus of a lying down position.

The second part of this sentence matches my experience: I foolishly tried saving the cost of a desk for a while by using my computer lying in bed. Eventually I bought that desk and it was one of the best investments I ever made: my ability to stop procrastinating and avoid distractions went through the roof as soon as I was upright.

Comment author: Yvain 23 December 2011 04:36:17AM *  1 point [-]

I'll be in Cincinnati on the 22nd. I may be able to make it a day early if this gets switched to there; otherwise I'd be happy to meet anyone there who wants to meet me.

Comment author: Yvain 16 December 2011 12:48:47AM *  5 points [-]
Comment author: Yvain 12 December 2011 03:29:44PM *  23 points [-]

Speaking of Romney, this Romney story from the Post (the homeless shelter one) struck me as an unusually pure example of a rationalist world-view in action.

Comment author: Eugine_Nier 12 December 2011 01:41:32AM -1 points [-]

And probably the most important reason is that medicine is weird. Even when the smartest people try to predict results that should be obvious, they very often get them wrong. "Based on what I know about the body, this sounds like it should work" is the worst reason to do anything. I know that sounds contrary to Bayes, but getting burned again and again by things that sound like they should work has recalibrated me on this one.

Reality isn't weird. What this means is that you know less about the body then you think you do.

Comment author: Yvain 12 December 2011 03:25:15PM *  9 points [-]

Well, "reality isn't weird" can mean a couple of different things. "Weird" is a two-part predicate like "sexiness"; things are only weird in reference to some particular mind's preconceptions. Even Yog-Sothoth doesn't seem weird to his own mother.

But if we use the word "weird" as a red flag to tell others that they can expect to be surprised or confused when entering a certain field, as long as we can predict that their minds and preconceptions work somewhat like ours, it's a useful word.

I think Eliezer's "reality is not weird" post was just trying to say that we can't blame reality for being weird, or expect things to be irreducibly weird even after we challenge our preconceptions. I don't think Eliezer was saying that we can't describe anything as "weird" if it actually exists; after all, he himself has been known to describe certain potential laws of physics as weird.

(man, basing an argument on the trivial word choices of a venerated community leader spotted in an old archive makes me feel so Jewish)

Comment author: buybuydandavis 12 December 2011 08:10:16AM *  -1 points [-]

Absent any other prior, why would you use anything other than "My body will react to hormones the same way most other people's bodies react to hormones"?

First, because I am not absent other informational priors. I have a lifetime of informational priors about my own body. I also have access to pubmed, wikipedia, my 23andme genomic data, my personal medical history, my family's medical history, and lab testing services that can take accurate measurements of me.

There are no clinical trials that have controlled for that information.

Second, because I know others are not identical to me. Basing my choices solely on some statistical outcome on a pool of patients where I have none of that kind of information, and indeed the doctors involved didn't take that information, and didn't factor it into their solutions for their patients, strikes me as throwing out most all of my relevant data and trusting the results produced by a blind man with a shotgun.

Moreover, refusing to experiment on yourself is to refuse to look at reality and take actual data about the system you're interested in - you. That's poor decisions theory, poor inference, and poor problem solving.

Yes, medicine is weird. Therefore, instead of thinking that you have it all worked out, or that a clinical trial has it all worked out for you, the rational thing to do is to evaluate options that might work, their costs and risks, try things, take measurements, update your model based on that additional data, and try again. Sure, if there are clinical trials, avail yourself of that information as well. Nice place to find candidate treatments. But you're deluded if you think a positive result means it will assuredly work for you, and you're deluded if you think a negative results, a "failure to reject", means it won't. At a minimum, if the trial didn't have a crossover study, it hasn't ruled out that the treatment is a perfect cure for some subset of people with the problem.

Any decent doctor I've had has basically said that all treatments are experiments for a particular person - maybe it will work for you, maybe not.

I don't know that I have incentives any different from anyone else with a malady. I wish to get better. I recognize that there are risks involved in the attempts to get better. What doctors fail to appreciate, probably because it's not really their problem, is that doing nothing also has a cost - the likely continuance of my malady.

We don't limit our pool of potential solutions to our problems to solutions "validated" by double blinded placebo controlled trials in any other aspect of life, because it isn't rational to do so. It's not rational for medical problems either.

Comment author: Yvain 12 December 2011 03:05:26PM *  9 points [-]

It's not about a positive result meaning something will "assuredly work for you". Only a Sith deals in absolutes. It's about cost-benefit analysis.

To give an example, no reasonable person would self-experiment to see if cyanide cures their rash. Although there's a distant probability your body has some wildly unusual reaction to cyanide in which it cures rashes, it's much more likely that cyanide will kill you, the same way it kills everyone else. Although it might be worth a shot if cyanide had no downside, we have very strong evidence that on average it has a very large downside.

The same is true of HRT. People were using it to improve their cardiovascular health. We found that, on average, it decreases cardiovascular health. You can still try using it on the grounds that it might paradoxically increase yours, but on average, you will lose utility.

Consider the analogy to a lottery. You have different numbers than everyone else does. Just because someone else lost the lottery with their numbers, doesn't mean you will lose the lottery with your numbers. But if we study all lottery participants for ten years and find that on average they lose money, then unless you have a specific reason to think your numbers are better than everyone else's (not just different), you should expect to lose money too.

Now things would be different with a treatment with no downside (like eating a lot of some kind of food, or taking a safe and cheap supplement) - as long as you don't mind the loss of time and money you can experiment all you want with those (though I still think you'd have trouble with bias and privileging the hypothesis, and that a rational person wouldn't find a lot of these harmless self-experiments worth the time and the money at all). And things would be different if the potential benefit and potential harm had different levels of utility for you: for example, if you wanted to cure your joint pain so badly you didn't mind risking heart attack as a side effect. I think this is what you're aiming at in your post above, and for those cases, I agree with you.

But when you're taking a treatment like HRT which is intended to prevent heart attacks, but actually on average increases heart attacks, then shut up and multiply.

Also, don't call it "self-experimentation" when you're talking about preventing cardiovascular disease, since you never end up with any usable self-data (as opposed to, say, self-experimenting with medication for joint pain, where you might get a strong result of your joint pain disappearing that you can trace with some confidence to the medication). Call it what it is - gambling.

Comment author: Yvain 11 December 2011 01:05:03PM *  50 points [-]

Walking on land is probably impossible, Pre-Cambrian researchers announced, since even if we did evolve some sort of "legs" our gills would be unable to extract oxygen from the environment.

Comment author: Yvain 11 December 2011 12:59:26PM 22 points [-]

Though keep in mind that it's much more complicated than "Haha, some idiots were drinking chemicals from horse urine but now we know better."

HRT is "undoubtedly" effective for hot flashes, osteoporosis, and atrophic vaginitis. It's probably effective against some cancers. It may be effective against heart disease and dementia when taken in the first decade after menopause, but not after (the women in the study you mention were mostly in their 60s). However, all of the risks mentioned here are very real and so HRT is not indicated except when people have premature menopause or unusually severe menopausal symptoms (these people should talk to their doctors about it)

Also, why title this "Why We Need Better Science"? Sounds to me like we have great science, we just need people to hold their horses (pun not intended) until the results come in instead of chasing after every therapeutic fad.

Comment author: buybuydandavis 11 December 2011 03:55:23AM -2 points [-]

I've been surprised at how much deference is given to randomized clinical trials here. Particularly for hormone replacement - it's bad inference, and bad decision theory.

Since when are Bayesians enamored of the results of trials against a null hypothesis that throws away almost all the relevant information about you, and leaves the decision about treatment for you up to the results of a 95% confidence interval on people who are not you, for whom you've also thrown away (or never collected) most of the relevant data?

Of course it depends on the nature of the malady you're trying to treat, but the best indicator of whether someone works for you is a trial on you. If the treatment is generally safe, try it and see. Measure relevant variables. See how it works. on the symptoms. People doing HRT generally aim to keep their levels within reference ranges, using the actual hormones that already exist in our bodies, instead of horse hormones. Too much or too little testosterone may be bad, but it's not like my body has never had to deal with it before.

Maybe it's just because I'm a little older than the mean here, but I don't think I have decades to wait for decades of longitudinal studies on life extension strategies, which probably won't even get decent funding until a significant mass of people have used them for a decade without any such studies to rely on. If I want to be around for escape velocity, I think I've got to get moving now. I don't use a 10 year old cel phone. I'm not going to limit myself to 10 year old medical technology either.

Comment author: Yvain 11 December 2011 12:46:17PM 14 points [-]

Absent any other prior, why would you use anything other than "My body will react to hormones the same way most other people's bodies react to hormones"?

And you can't self-experiment on risk of a heart attack. Your only endpoint is "I had a heart attack" or "I didn't have a heart attack", and even if you don't mind getting your experimental result exactly one instant too late to help you, with a sample size of one you can't draw any conclusions about whether taking HRT for ten years contributed to your heart attack or not.

And probably the most important reason is that medicine is weird. Even when the smartest people try to predict results that should be obvious, they very often get them wrong. "Based on what I know about the body, this sounds like it should work" is the worst reason to do anything. I know that sounds contrary to Bayes, but getting burned again and again by things that sound like they should work has recalibrated me on this one.

If you're saying that you have unusual incentives here - eg that you value the possibility of adding to your natural lifespan enough that you're willing to accept a small risk of subtracting from it and a large risk that you're wasting time and money, that's fair enough.

In response to [POLL] Year survey
Comment author: lavalamp 09 December 2011 08:12:46PM *  4 points [-]

Gah... you bastard.

Edit: I used a random number generator. I hope that's not cheating.

Comment author: Yvain 09 December 2011 10:52:42PM 6 points [-]

Why would you do that? If you know this is set up to detect patterns in how humans pick random numbers, isn't specifically entering a data point in order to thwart that sort of like providing fake data on a survey to mess with the results?

In response to 2011 Survey Results
Comment author: Yvain 04 December 2011 07:14:42PM *  40 points [-]

Running list of changes for next year's survey:

  1. Ask who's a poster versus a lurker!
  2. A non-write-in "Other" for most questions
  3. Replace "gender" with "sex" to avoid complaints/philosophizing.
  4. Very very clear instructions to use percent probabilities and not decimal probabilities
  5. Singularity year question should have explicit instructions for people who don't believe in singularity
  6. Separate out "relationship status" and "looking for new relationships" questions to account for polys
  7. Clarify that research is allowed on the probability questions
  8. Clarify possible destruction of humanity in cryonics/antiagathics questions.
  9. What does it mean for aliens to "exist in the universe"? Light cone?
  10. Make sure people write down "0" if they have 0 karma.
  11. Add "want to sign up, but not available" as cryonics option.
  12. Birth order.
  13. Have children?
  14. Country of origin?
  15. Consider asking about SAT scores for Americans to have something to correlate IQs with.
  16. Consider changing morality to PhilPapers version.
In response to comment by Yvain on 2011 Survey Results
Comment author: Yvain 07 December 2011 01:11:27PM 3 points [-]

Everyone who's suggesting changes: you are much more likely to get your way if you suggest a specific alternative. For example, instead of "handle politics better", something like "your politics question should have these five options: a, b, c, d, and e." Or instead of "use a more valid IQ measure", something more like "Here's a site with a quick and easy test that I think is valid"

In response to comment by Larks on 2011 Survey Results
Comment author: Vladimir_Nesov 05 December 2011 04:27:05PM 2 points [-]

Are the questions for the 2009 survey available somewhere?

Comment author: Yvain 05 December 2011 08:21:01PM 3 points [-]
In response to 2011 Survey Results
Comment author: wedrifid 05 December 2011 03:36:10AM 12 points [-]

So we can only prove that 519 people post on Less Wrong.

Where by 'prove' we mean 'somebody implied that they did on an anonymous online survey'. ;)

Comment author: Yvain 05 December 2011 03:16:27PM 13 points [-]

You mean, as opposed to that kind of proof where we end up with a Bayesian probability of exactly one? :)

2011 Survey Results

94 Post author: Yvain 05 December 2011 10:49AM

A big thank you to the 1090 people who took the second Less Wrong Census/Survey.

Does this mean there are 1090 people who post on Less Wrong? Not necessarily. 165 people said they had zero karma, and 406 people skipped the karma question - I assume a good number of the skippers were people with zero karma or without accounts. So we can only prove that 519 people post on Less Wrong. Which is still a lot of people.

I apologize for failing to ask who had or did not have an LW account. Because there are a number of these failures, I'm putting them all in a comment to this post so they don't clutter the survey results. Please talk about changes you want for next year's survey there.

Of our 1090 respondents, 972 (89%) were male, 92 (8.4%) female, 7 (.6%) transexual, and 19 gave various other answers or objected to the question. As abysmally male-dominated as these results are, the percent of women has tripled since the last survey in mid-2009.

We're also a little more diverse than we were in 2009; our percent non-whites has risen from 6% to just below 10%. Along with 944 whites (86%) we include 38 Hispanics (3.5%), 31 East Asians (2.8%), 26 Indian Asians (2.4%) and 4 blacks (.4%).

Age ranged from a supposed minimum of 1 (they start making rationalists early these days?) to a more plausible minimum of 14, to a maximum of 77. The mean age was 27.18 years. Quartiles (25%, 50%, 75%) were 21, 25, and 30. 90% of us are under 38, 95% of us are under 45, but there are still eleven Less Wrongers over the age of 60. The average Less Wronger has aged about one week since spring 2009 - so clearly all those anti-agathics we're taking are working!

In order of frequency, we include 366 computer scientists (32.6%), 174 people in the hard sciences (16%) 80 people in finance (7.3%), 63 people in the social sciences (5.8%), 43 people involved in AI (3.9%), 39 philosophers (3.6%), 15 mathematicians (1.5%), 14 statisticians (1.3%), 15 people involved in law (1.5%) and 5 people in medicine (.5%).

48 of us (4.4%) teach in academia, 470 (43.1%) are students, 417 (38.3%) do for-profit work, 34 (3.1%) do non-profit work, 41 (3.8%) work for the government, and 72 (6.6%) are unemployed.

418 people (38.3%) have yet to receive any degrees, 400 (36.7%) have a Bachelor's or equivalent, 175 (16.1%) have a Master's or equivalent, 65 people (6%) have a Ph.D, and 19 people (1.7%) have a professional degree such as an MD or JD.

345 people (31.7%) are single and looking, 250 (22.9%) are single but not looking, 286 (26.2%) are in a relationship, and 201 (18.4%) are married. There are striking differences across men and women: women are more likely to be in a relationship and less likely to be single and looking (33% men vs. 19% women). All of these numbers look a lot like the ones from 2009.

27 people (2.5%) are asexual, 119 (10.9%) are bisexual, 24 (2.2%) are homosexual, and 902 (82.8%) are heterosexual.

625 people (57.3%) described themselves as monogamous, 145 (13.3%) as polyamorous, and 298 (27.3%) didn't really know. These numbers were similar between men and women.

The most popular political view, at least according to the much-maligned categories on the survey, was liberalism, with 376 adherents and 34.5% of the vote. Libertarianism followed at 352 (32.3%), then socialism at 290 (26.6%), conservativism at 30 (2.8%) and communism at 5 (.5%).

680 people (62.4%) were consequentialist, 152 (13.9%) virtue ethicist, 49 (4.5%) deontologist, and 145 (13.3%) did not believe in morality.

801 people (73.5%) were atheist and not spiritual, 108 (9.9%) were atheist and spiritual, 97 (8.9%) were agnostic, 30 (2.8%) were deist or pantheist or something along those lines, and 39 people (3.5%) described themselves as theists (20 committed plus 19 lukewarm)

425 people (38.1%) grew up in some flavor of nontheist family, compared to 297 (27.2%) in committed theist families and 356 in lukewarm theist families (32.7%). Common family religious backgrounds included Protestantism with 451 people (41.4%), Catholicism with 289 (26.5%) Jews with 102 (9.4%), Hindus with 20 (1.8%), Mormons with 17 (1.6%) and traditional Chinese religion with 13 (1.2%)

There was much derision on the last survey over the average IQ supposedly being 146. Clearly Less Wrong has been dumbed down since then, since the average IQ has fallen all the way down to 140. Numbers ranged from 110 all the way up to 204 (for reference, Marilyn vos Savant, who holds the Guinness World Record for highest adult IQ ever recorded, has an IQ of 185).

89 people (8.2%) have never looked at the Sequences; a further 234 (32.5%) have only given them a quick glance. 170 people have read about 25% of the sequences, 169 (15.5%) about 50%, 167 (15.3%) about 75%, and 253 people (23.2%) said they've read almost all of them. This last number is actually lower than the 302 people who have been here since the Overcoming Bias days when the Sequences were still being written (27.7% of us).

The other 72.3% of people who had to find Less Wrong the hard way. 121 people (11.1%) were referred by a friend, 259 people (23.8%) were referred by blogs, 196 people (18%) were referred by Harry Potter and the Methods of Rationality, 96 people (8.8%) were referred by a search engine, and only one person (.1%) was referred by a class in school.

Of the 259 people referred by blogs, 134 told me which blog referred them. There was a very long tail here, with most blogs only referring one or two people, but the overwhelming winner was Common Sense Atheism, which is responsible for 18 current Less Wrong readers. Other important blogs and sites include Hacker News (11 people), Marginal Revolution (6 people), TV Tropes (5 people), and a three way tie for fifth between Reddit, SebastianMarshall.com, and You Are Not So Smart (3 people).

Of those people who chose to list their karma, the mean value was 658 and the median was 40 (these numbers are pretty meaningless, because some people with zero karma put that down and other people did not).

Of those people willing to admit the time they spent on Less Wrong, after eliminating one outlier (sorry, but you don't spend 40579 minutes daily on LW; even I don't spend that long) the mean was 21 minutes and the median was 15 minutes. There were at least a dozen people in the two to three hour range, and the winner (well, except the 40579 guy) was someone who says he spends five hours a day.

I'm going to give all the probabilities in the form [mean, (25%-quartile, 50%-quartile/median, 75%-quartile)]. There may have been some problems here revolving around people who gave numbers like .01: I didn't know whether they meant 1% or .01%. Excel helpfully rounded all numbers down to two decimal places for me, and after a while I decided not to make it stop: unless I wanted to do geometric means, I can't do justice to really small grades in probability.

The Many Worlds hypothesis is true: 56.5, (30, 65, 80)
There is intelligent life elsewhere in the Universe: 69.4, (50, 90, 99)
There is intelligent life elsewhere in our galaxy: 41.2, (1, 30, 80)
The supernatural (ontologically basic mental entities) exists: 5.38, (0, 0, 1)
God (a supernatural creator of the universe) exists: 5.64, (0, 0, 1)
Some revealed religion is true: 3.40, (0, 0, .15)
Average person cryonically frozen today will be successfully revived: 21.1, (1, 10, 30)
Someone now living will reach age 1000: 23.6, (1, 10, 30)
We are living in a simulation: 19, (.23, 5, 33)
Significant anthropogenic global warming is occurring: 70.7, (55, 85, 95)
Humanity will make it to 2100 without a catastrophe killing >90% of us: 67.6, (50, 80, 90)

There were a few significant demographics differences here. Women tended to be more skeptical of the extreme transhumanist claims like cryonics and antiagathics (for example, men thought the current generation had a 24.7% chance of seeing someone live to 1000 years; women thought there was only a 9.2% chance). Older people were less likely to believe in transhumanist claims, a little less likely to believe in anthropogenic global warming, and more likely to believe in aliens living in our galaxy. Community veterans were more likely to believe in Many Worlds, less likely to believe in God, and - surprisingly - less likely to believe in cryonics (significant at 5% level; could be a fluke). People who believed in high existential risk were more likely to believe in global warming, more likely to believe they had a higher IQ than average, and more likely to believe in aliens (I found that same result last time, and it puzzled me then too.)

Intriguingly, even though the sample size increased by more than 6 times, most of these results are within one to two percent of the numbers on the 2009 survey, so this supports taking them as a direct line to prevailing rationalist opinion rather than the contingent opinions of one random group.

Of possible existential risks, the most feared was a bioengineered pandemic, which got 194 votes (17.8%) - a natural pandemic got 89 (8.2%), making pandemics the overwhelming leader. Unfriendly AI followed with 180 votes (16.5%), then nuclear war with 151 (13.9%), ecological collapse with 145 votes (12.3%), economic/political collapse with 134 votes (12.3%), and asteroids and nanotech bringing up the rear with 46 votes each (4.2%).

The mean for the Singularity question is useless because of the very high numbers some people put in, but the median was 2080 (quartiles 2050, 2080, 2150). The Singularity has gotten later since 2009: the median guess then was 2067. There was some discussion about whether people might have been anchored by the previous mention of 2100 in the x-risk question. I changed the order after 104 responses to prevent this; a t-test found no significant difference between the responses before and after the change (in fact, the trend was in the wrong direction).

Only 49 people (4.5%) have never considered cryonics or don't know what it is. 388 (35.6%) of the remainder reject it, 583 (53.5%) are considering it, and 47 (4.3%) are already signed up for it. That's more than double the percent signed up in 2009.

231 people (23.4% of respondents) have attended a Less Wrong meetup.

The average person was 37.6% sure their IQ would be above average - underconfident! Imagine that! (quartiles were 10, 40, 60). The mean was 54.5% for people whose IQs really were above average, and 29.7% for people whose IQs really were below average. There was a correlation of .479 (significant at less than 1% level) between IQ and confidence in high IQ.

Isaac Newton published his Principia Mathematica in 1687. Although people guessed dates as early as 1250 and as late as 1960, the mean was...1687 (quartiles were 1650, 1680, 1720). This marks the second consecutive year that the average answer to these difficult historical questions has been exactly right (to be fair, last time it was the median that was exactly right and the mean was all of eight months off). Let no one ever say that the wisdom of crowds is not a powerful tool.

The average person was 34.3% confident in their answer, but 41.9% of people got the question right (again with the underconfidence!). There was a highly significant correlation of r = -.24 between confidence and number of years error.

This graph may take some work to read. The x-axis is confidence. The y-axis is what percent of people were correct at that confidence level. The red line you recognize as perfect calibration. The thick green line is your results from the Newton problem. The black line is results from the general population I got from a different calibration experiment tested on 50 random trivia questions; take the intercomparability of the two with a grain of salt.

As you can see, Less Wrong does significantly better than the general population. However, there are a few areas of failure. First is that, as usual, people who put zero and one hundred percent had nonzero chances of getting the question right or wrong: 16.7% of people who put "0" were right, and 28.6% of people who put "100" were wrong (interestingly, people who put 100 did worse than the average of everyone else in the 90-99 bracket, of whom only 12.2% erred). Second of all, the line is pretty horizontal from zero to fifty or so. People who thought they had a >50% chance of being right had excellent calibration, but people who gave themselves a low chance of being right were poorly calibrated. In particular, I was surprised to see so many people put numbers like "0". If you're pretty sure Newton lived after the birth of Christ, but before the present day, that alone gives you a 1% chance of randomly picking the correct 20-year interval.

160 people wanted their responses kept private. They have been removed. The rest have been sorted by age to remove any information about the time they took the survey. I've converted what's left to a .xls file, and you can download it here.

In response to 2011 Survey Results
Comment author: Morendil 04 December 2011 10:42:35PM 7 points [-]

I am officially very surprised at how many that is. Also officially, poorly calibrated at both the 50% (no big deal) and the 90% (ouch, ouch, ouch) confidence levels.

Comment author: Yvain 04 December 2011 10:48:07PM 4 points [-]

You're okay. I asked the question about the number of responses then. When I asked the question, there were only 970 :)

In response to 2011 Survey Results
Comment author: J_Taylor 04 December 2011 08:35:47PM 1 point [-]

I have no idea if this is universal. (Probably not.) However, in my area, using the term "blacks" in certain social circles is not considered proper vocabulary.

I don't have any huge problem with using the term. However, using it may be bad signalling and leaves Lesswrong vulnerable to pattern-matching.

Comment author: Yvain 04 December 2011 09:15:54PM 10 points [-]

What would you prefer? "Blacks" is the way I've seen it used in medical and psychological journal articles.

In response to 2011 Survey Results
Comment author: Unnamed 04 December 2011 07:48:02PM 2 points [-]

Could you make a copy of the survey (with the exact wordings of all the questions) available for download?

Comment author: Yvain 04 December 2011 08:40:35PM 1 point [-]

I've re-opened the survey at https://docs.google.com/spreadsheet/viewform?formkey=dHlYUVBYU0Q5MjNpMzJ5TWJESWtPb1E6MQ , but please don't send in any more responses.

In response to 2011 Survey Results
Comment author: Yvain 04 December 2011 07:14:42PM *  40 points [-]

Running list of changes for next year's survey:

  1. Ask who's a poster versus a lurker!
  2. A non-write-in "Other" for most questions
  3. Replace "gender" with "sex" to avoid complaints/philosophizing.
  4. Very very clear instructions to use percent probabilities and not decimal probabilities
  5. Singularity year question should have explicit instructions for people who don't believe in singularity
  6. Separate out "relationship status" and "looking for new relationships" questions to account for polys
  7. Clarify that research is allowed on the probability questions
  8. Clarify possible destruction of humanity in cryonics/antiagathics questions.
  9. What does it mean for aliens to "exist in the universe"? Light cone?
  10. Make sure people write down "0" if they have 0 karma.
  11. Add "want to sign up, but not available" as cryonics option.
  12. Birth order.
  13. Have children?
  14. Country of origin?
  15. Consider asking about SAT scores for Americans to have something to correlate IQs with.
  16. Consider changing morality to PhilPapers version.
Comment author: Nisan 04 December 2011 04:42:00PM 5 points [-]

:D

The music is I am my own grandpa, right?

Comment author: Yvain 04 December 2011 07:02:37PM *  5 points [-]

It is. Mike edited it a little to include some more choruses.

Comment author: Yvain 04 December 2011 01:00:24PM *  15 points [-]

I don't think your point applies to this specific graph, since it is a cumulative odds ratio graph selected for initially high error in one direction.

Associations from Ioannides' analysis were selected for inclusion on this graph because there were dramatic changes from the first study to the next studies - in this case, because the first study had a high effect size and the others showed lower effect sizes. Studies were selected for Ioannides' analysis in the first place because there were meta-analyses around them - which means lots of people tried to replicate them - which means the initial result must have been surprising and interesting. So these have been double-selected already for original studies likely to have high errors.

In a cumulative odds ratio graph (unlike the individual odds ratio graphs of the classic funnel plot), most of the work of subsequent studies will go to bringing the trend line closer to the mean. Even a study that shows an effect in the opposite direction as the original won't move the trend line to the other side of the identity line if the effect is smaller. So if the graph is arranged so that the most surprising and deviant result is the first, then it could very well look like this one even if there were no publication bias.

This is doubly true when the graphs are not actually converging to one - Ioannides' paper admits that three of these are probably real associations, and this is most obvious here on the DRD2 line, which converges around .5.

Comment author: Nisan 30 November 2011 04:12:51AM 4 points [-]

Ah, that makes sense.

Yes, please :D

Comment author: Yvain 03 December 2011 07:22:22PM *  13 points [-]

WE DIDN'T START THE FIRE (GRAND FINALE)

Thales, Socrates, Zeno and Empedocles
'Naximander, 'Naximenes, Anaxagoras
Logos eros ontos noos, square of the hypotenuse
Echecrates, Archelaus, and Protagoras
Gorgias, Parmenides, Strato and Diogenes
Euclid writes his Elements, Pyrrho says he's not convinced
Aristotle, Plato, Cicero and Cato
Plotinus, Apollodorus, come on baby sing the chorus

[Chorus: We didn't start the fire!
It was always burning since the dawn of learning!
We didn't start the fire!
Each new sage and writer makes it even brighter!
]

Augustine, Boethius, Erigena Scotus
Avicenna, Avicebron, Al-Ghalazi
Gregory, St. Bernard, Heloise and Abelard
Terms exist, Eucharist, Christ's divinity
Necessary entities, Moses ben Maimonides
Anselm's ontological, Summa Theologica
Occam, Buridan, Aristotle's back again
Angels, free will, origin of evil!

[Chorus]

Renaissance, The Prince, Luther founds the Protestants
Earth goes round the sun, Calvin and Melancthon
Telescopes, More's high hopes, mystics and reformist popes
Galileo, Giordano Bruno

Cogito ergo sum, Francis Bacon, deism
Jesuits, Mersenne, Sweden has a Catholic queen
Hobbes, Locke, Cherbury, tangents, probability
Natural law, pumping air, Newton lays the heavens bare!

[Chorus]

Bishop Berkeley, Leibnitz, who discovered calculus?
Voltaire, David Hume, Malthus fears impending doom
Adam Smith, dogmatic slumber, greatest good for greatest number
Mill, Hegel, Kierkegaard, Darwin hits religion hard
Comte, James, Schopenhauer, Karl Marx, will to power
Here's a hand, plain as day, what else do I have to say!

[Chorus]

Freud, Jung, morning star, Russell's new Principia
Carnap wants concreteness, Godel's incompleteness
Tarski, Popper, dasein, late and early Wittgenstein
Ryle, Turing, Sartre thinks life is boring
Ayer, Quine, rights for blacks, trolley car is off its tracks
'Pologetics, new aesthetics, Rawls, Searle, cybernetics
Chomsky, Rorty, Nagel's bats, modal logic, brains in vats
Virtue ethics redux - preach it baby all is flux!

We didn't start the fire!
It was always burning, since the dawn of learning!
We didn't start the fire!
But the more we know, the brighter it will glow...
...and grow...and grow...and grow...

[Chorus]

Comment author: Nisan 30 November 2011 04:12:51AM 4 points [-]

Ah, that makes sense.

Yes, please :D

Comment author: Yvain 03 December 2011 06:58:25PM *  14 points [-]

WHAT IS IT LIKE TO BE A BAT

Many many years ago, when I was but a youth
I studied to all hours in a frantic search for Truth
I was sitting in my study, 'midst the books that I had read
When I glanced out of the window, and a bat flew overhead
And I thought:

What is it like to be a bat?
Oh, if only I knew that!
I could probably explain
All the secrets of the brain
If I knew what it was like to be a bat

It seemed to me the answer to the riddle of the mind
Would be found in the experience of those least like humankind
In unfamiliar qualia, like those of the bat who hears
The echo of an insect in its ultrasonic ears
And I thought:

What is it like to be a bat?
Oh, if only I knew that!
I could be more self-aware
I could become department chair!
If I knew what it was like to be a bat

It turned into a passion, then an all-consuming mania
My research took me far and wide, till I reached Transylvania
And there beneath a castle tossed by storms and lost in mist
The townspeople declared the local noble could assist
Could tell me:

What it was like to be a bat?
Oh, if only I knew that!
I could see with different eyes
I could win a Nobel Prize!
If I knew what it was like to be a bat

The noble said he'd help me, and I babbled out my thanks
He responded with a smile, and it looked a lot like fangs
I really don't remember much that happened after that
But when I awoke that evening, I could turn into a bat.

What is it like to be a bat?
Oh, I finally know that!
Now I sleep within a coffin
And don't see the sun too often
But I know what it's like to be a bat

What is it like to be a bat?
Oh, I'm an expert on that
I was bookish once, and stuffy
Now I live in fear of Buffy
But I know what it's like to be a bat

Comment author: Nisan 30 November 2011 04:12:51AM 4 points [-]

Ah, that makes sense.

Yes, please :D

Comment author: Yvain 03 December 2011 06:53:58PM *  11 points [-]

ZOMBIES' SECRET

The soul is an illusion
A cause of much confusion
And very little gain
And the facts would be consistent
If my soul were nonexistent
If I only had a brain

I'd unravel schemes Cartesian
Replacing them with reashon
And make things clear again
I'd elucidate how mostly
I'm a shell, but not so ghostly
If I only had a brain

Oh I
Can't tell you why
So many folk expound
On mental states they don't know how to ground
When no such states
Are to be found!

I'd abandon all reliance
On fields less pure than science
Confusing in the main
Talk of mind would be denotin'
Simple acts of cells and prot'in
If I only had a brain

Comment author: Yvain 03 December 2011 01:58:48PM 6 points [-]

Your summary is that there might be a difference between students and adults here, but isn't the last paragraph of the quoted text saying that they considered this hypothesis, but that students were also non-hyperbolic when they tested them under the same protocol?

Comment author: Yvain 03 December 2011 03:26:56AM *  1 point [-]

New experiment supports evopsych idea that some out-group prejudice is related to disease risk (though I wish it had been controlled with a state of generalized non-disease stress to see whether it's just stress that increases prejudice).

Comment author: Yvain 03 December 2011 02:08:56AM 7 points [-]

This survey is now closed. I'll have data eventually.

The answer to my question from November 12 was 970 people.

Comment author: endoself 27 November 2011 07:13:25PM 6 points [-]

But his exposition is marred by a series of incongruities, including his repeated mispronunciations of his own name ("All is Water", lyrics only).

I read all the other words the wrong way instead. I think it works well that way.

Comment author: Yvain 29 November 2011 09:56:00PM 5 points [-]

Yes, well, the whole thing originally came from me writing the lyrics before I knew how to pronounce "Thales" correctly.

Comment author: lukeprog 28 November 2011 04:29:56AM 2 points [-]

"Zombies' Secret" song link is broken.

Comment author: Yvain 29 November 2011 09:55:19PM 2 points [-]

Think I fixed it; correct link should be http://www.youtube.com/watch?v=OFbE87P82-w

Comment author: Nisan 29 November 2011 04:18:40AM *  14 points [-]

Here's the lyrics to Yvain's The Devil and Blaise Pascal:

The Devil went down to Paris.
He was looking for a soul to steal.
He saw Blaise Pascal hunched over a scroll
And he said, "Let's make a deal.

"I've heard you play the fiddle.
Well, I'm a fiddle player too.
So I bet your soul against a fiddle of gold
'Cause I think I'm better than you.

Said Pascal, "Monsieur le Diable,
I can sing and I can dance,
And I'm better than you, yes I know it's true,
Ninety-nine-percent-plus chance.

"And that fiddle of gold, well if it was sold,
It could feed me pretty well.
But there's infinity disutility
Out of any chance of hell.

"So my calculations tell me
That my answer must be non.
Now take your poison deal,
Say au revoir and get thee gone."

Pascal, set your gains as X and make your losses Y.
Estimate your chance of winning and cross-mulitply.
On one side, P-sub-win times value(fiddle made of gold),
And on the other, P-sub-loss times soul.

Well, smoke comes out the Devil's ears,
But in his heart he knows he's beat.
Since he can't get a soul, he switches his goal,
'Cause revenge to him is sweet.

And he says, "I've been outwitted, sure,
And I guess I'm leaving soon.
But I'm still your guest and I've one request:
Let me fiddle you a tune."

The Devil puts down his fiddle,
And now Pascal takes up his.
And he plays a tune he's been working on;
It sounds a lot like this:

Fire on the mountain, run boys run!
One
One one
One two one
One! / Three three one
One four six four! / One
one five and fiddle some more!

Said the Devil, "It's infinitely bad
To end that song and dance.
And I won't tell you why, and I'd probably lie,
But can you really take that chance?"

Blaise fills with trepidation
As his calculations turn the Devil's way,
And they say in the Paris catacombs
His ghost is fiddling to this day.

Fire on the mountain, run boys run!
One
One one
One two one
One! / Three three one
One four six four! / One
one five and fiddle some more!

Comment author: Yvain 29 November 2011 09:53:56PM 6 points [-]

"will tell you why" should be "won't tell you why".

Do you want the other song lyrics?

Comment author: orthonormal 26 November 2011 07:03:59AM *  19 points [-]

This is one of the best things I've ever seen.

ETA: Seriously, if I were a selfish billionaire (with my current aesthetic tastes), I would set up a seven-figure trust fund for you and make a similarly large donation to your preferred charity, on the sole condition that you drop medicine and take up writing and other creative endeavors exclusively.

Comment author: Yvain 29 November 2011 09:52:51PM *  6 points [-]

Thanks! That must be one of the best compliments I've ever gotten.

King Under The Mountain: Adventure Log + Soundtrack

56 Yvain 25 November 2011 10:29PM

With the help of many dedicated Less Wrongers (players muflax, Karl, Charlie, and Emile; musicians Mike Blume and Alicorn, technical support Ari Rahikkala) we have successfully completed what is, as far as I know, the first actual Dungeons and Discourse adventure anywhere. Except we're not calling it that, because I don't have the rights to use that name. Though it's not precisely rationality related, I hope it is all right if I post a summary of the adventure by popular demand.

Also, at some point it turned into a musical. The first half of the songs are only available as lyrics at the moment, but Alicorn and MBlume very kindly produced the second half as real music, which I've uploaded to YouTube and linked at the bottom of this post (skip to it now).

THE ADVENTURE

BACKGROUND

The known world has many sects and religions, but all contain shadowy legends of two primeval deities: Sophia, Goddess of Wisdom; and Aleithos, God of Truth. When Sophia announced her plan to create thinking, rational beings, Aleithos objected, declaring that they would fall into error and produce endless falsehoods. Sophia ignored her brother's objections and created humankind, who named the world after their goddess-mother. But Aleithos' fears proved well-founded: humankind fell into error and produced endless falsehoods, and their clamor drove the God of Truth insane.

The once mighty Aleithos fell from heaven, and all of his angelic servants turned into Paradox Beasts, arachnoid monstrosities that sought and devoured those who challenged the laws of logic. Over centuries, most of the Paradox Beasts were banished, but Aleithos himself remained missing. And though thousands of seekers set off to all the corners of the world in search of Truth, the Mad God keeps his own counsel, if He still even exists at all.

The Truth God's madness had one other effect; the laws of physics, once inviolable, turned fluid, and those sufficiently advanced in the study of Truth gained apparently magical abilities. With knowledge literally being power, great philosophers and scientists built mighty cities and empires.

In the middle of the Cartesian Plain at the confluence of the rivers Ordinate and Abcissa stands the mightiest of all, the imperial city of Origin. At the very center of the city stands the infinitely tall Z-Axis Tower, on whose bottom floor lives the all-seeing Wizard of 0=Z. Surrounding the Tower are a host of colleges and universities that attract the greatest scholars from all over Origin, all gathered in service to the great project to find Truth.

Into the city comes Lady Cerune Russell, an exotic noblewoman from far-off parts seeking great thinkers to join her on a dangerous adventure. Four scholars flock to her banner. Nomophilos the Elder the Younger (Emile) is a political scientist studying the central role of laws in creating a just society. Phaidros (muflax) is a zealous Protestant theologian trying to meld strains of thought as disparate as Calvinism, Gnosticism, and W.L. Craig's apologetics. Ephraim (Charlie) is a Darwinian biologist with strong leftist sympathies and an experimental streak that sometimes gets him in trouble. And Macx (Karl) is a quiet but very precise logician with a talent for puzzles.

Cerune explains to the Original scholars that she is the last living descendant of Good King Bertrand, historic ruler of the land of Russellia far to the west. Russellia was the greatest nation in the world until two hundred years ago, when a cataclysm destroyed the entire kingdom in a single day and night. Now the skies above Russellia are dark and filled with choking ash, monsters roam its plains, and the Good King is said to be locked in a magical undying sleep deep beneath the Golden Mountain in the kingdom's center. Though many have traveled to Russellia in search of answers, none have returned alive; Cerune, armed with secret information from the Turing Oracle which she refuses to share, thinks she can do better. The four Originals agree to protect her as she makes the dangerous journey to the Golden Mountain to investigate the mysterious disaster and perhaps lift the curse. Cerune gives them a day in Origin to prepare for the journey.

CHAPTER ONE: ORIGIN

The party skip the city's major attractions, including the Z-Axis Tower and the Hagia Sophia, in favor of more academic preparations: a visit to the library to conduct research, and a shopping trip to Barnes & Aristoi Booksellers, where they purchase reading material for the journey ahead. Here, they find a map of the lands on the road to Russellia, including the unpleasant-sounding Slough of Despotism and the Shadow City of Xar-Morgoloth, whose very name inexplicably chills the air when spoken aloud. After a long discussion on how this thermodynamic-defying effect could probably be used to produce unlimited free energy, they return to more immediate matters and head to the armory to pick up some weapons - a trusty isoceles triangle for Nomophilos, a bow for Macx - before the stores close for the evening. After a final night in Origin, they meet Cerune at the city gates and set off.

They originally intend to stick to the course of the Abcissa, but it is flooding its banks and Cerune recommends crossing the river into Platonia at the Pons Asinorum. After being attacked by a Euclidean Elemental charged with letting no one enter who does not know geometry, they reach the other bank and find a strange old man, raving incomprehensibly. His turns of phrase start to make sense only after the party realizes that he is speaking as if he - and all objects - have no consistent identity.

In his roundabout way, he identifies himself as Heraclitus, the Fire Mage, one of the four great Elemental Mages of Platonia. Many years ago, he crossed into Origin on some errand, only to be ambushed by his arch-enemy, the Water Mage Thales. Thales placed a curse on Heraclitus that he could never cross the same river twice, trapping him on the wrong side of the Abcissa and preventing his return to Platonia. In order to dispel the curse, Heraclitus finds a loophole in the curse: he convinces himself that objects have no permanent identity, and so he can never cross the same river twice since it is not the same river and he is not the same man. Accepting this thesis, he crosses the Abcissa without incident - only to find that his new metaphysics of identity prevents him from forming goals, executing long-term plans, or doing anything more complicated than sitting by the riverbank and eating the fish that swim by.

This sets off a storm of conversation, as each member of the party tries to set Heraclitus right in their own way; Phaidros by appealing to God as a final arbiter of identity, Macx and Nomophilos by arguing that duty is independent of identity and that Heraclitus has a duty to his family and followers. Unfortunately, they make a logical misstep and end out convincing Heraclitus that it is illogical from his perspective to hold conversation; this ends the debate. And as the five philosophers stand around discussing what to do, they are ambushed by a party of assassins, who shoot poisoned arrows at them from a nearby knoll.

Outnumbered and outflanked, the situation seems hopeless, until Macx notices several of the attackers confused and unwilling to attack. With this clue, he identifies them as Buridan's Assassins, who in the presence of two equally good targets will hesitate forever, unable to choose: he yells to his friends to stand with two or more adventurers equidistant from each assassin, and sure enough, this paralyzes the archers and allows the party some breathing space.

But when a second group of assassins arrives to join the first, the end seems near - until Heraclitus, after much pondering, decides to accept his interlocutors' arguments for object permanence and joins in the battle. His fire magic makes short work of the remaining assassins, and when the battle is over, he thanks them and gives a powerful magic item as a gift to each. Then he disappears in a burst of flame after warning his new friends to beware the dangers ahead.

The party searches the corpses of the assassins - who all carry obsidian coins marked PLXM - and then camp for the night on the fringe of the Slough of Despotism.

CHAPTER TWO: THE SLOUGH OF DESPOTISM


The Slough of Despotism is a swamp unfortunately filled with allegators, giant reptiles who thrive on moral superiority and on casting blame. They accuse our heroes of trespassing on their property; our heroes counter that the allegators, who do not have a state to enforce property rights, cannot have a meaningful concept of property. The allegators threaten to form a state, but before they can do so the party manages to turn them against each other by pointing out where their property rights conflict; while the allegators argue, the adventurers sneak off.

They continue through the swamp, braving dense vegetation, giant snakes, and more allegators (who are working on the whole state thing; the party tells them that they're too small and disorganized to be a real state, and that they would have to unite the entire allegator nation under a mutually agreed system of laws) before arriving at an old barrow tomb. Though four of the five adventurers want to leave well enough alone, Ephraim's experimental spirits gets the better of him, and he enters the mound. Its local Barrow Wight has long since departed, but he has left behind a suit of Dead Wight Mail, which confers powerful bonuses on Conservatives and followers of the Right-Hand Path. Nomophilos, the party's Conservative, is all set to take the Mail when Phaidros objects that it is morally wrong to steal from the dead; this sparks a fight that almost becomes violent before Nomo finally backs down; with a sigh of remorse, he leaves the magic item where he found it.

Beyond the barrow tomb lies the domain of the Hobbesgoblins, the mirror image of the Allegators in that they have a strong - some might say dictatorial - state under the rule of their unseen god-king, Lord-Over-All. They are hostile to any foreigners who refuse to swear allegiance to their ruler, but after seeing an idol of the god-king - a tentacled monstrosity bearing more than a passing resemblance to Cthulhu - our heroes are understandably reluctant to do so. As a result, the Hobbesgoblins try to refuse them passage through their capital city of Malmesbury on the grounds that, without being subordinated to Lord-Over-All or any other common ruler, the adventurers are in a state of nature relative to the Hobbesgoblins and may rob, murder, or otherwise exploit them. The Hobbesgoblins don't trust mere oaths or protestations of morality - but Nomophilos finally comes up with a compromise that satisfies them. He offers them a hostage in return for their good behavior, handing them his pet tortoise Xeno. This satisfies the Hobbesgoblins as assurance of their good behavior, and the party passes through Malmesbury without incident.

On the far side of Malmesbury they come to a great lake, around which the miasmas of the swamp seem to swirl expectantly. On the shore of the lake lives Theseus with his two ships. Theseus tells his story: when he came of age, he set off on a trading expedition upon his father's favorite ship. His father made him swear to return the ship intact, but after many years of travel, Theseus realized that every part of the ship had been replaced and repaired, so that there was not a single piece of the ship that was the same as when it had left port. Mindful of his oath, he hunted down the old pieces he had replaced, and joined them together into a second ship. But now he is confused: is it the first or the second ship which he must return to his father?

The five philosophers tell Theseus that it is the first ship: the ship's identity is linked to its causal history, not to the matter that composes it. Delighted with this answer, he offers the second ship to the adventurers, who sail toward the far shore.

Halfway across the lake, they meet an old man sitting upon a small island. He introduces himself as Thomas Hobbes, and says that his spies and secret police have told him everything about the adventurers since they entered the Slough. Their plan to save Russellia is a direct threat to his own scheme to subordinate the entire world under one ruler, and so he will destroy them. When the party expresses skepticism, his "island" rises out of the water and reveals itself to be the back of the monstrous sea creature, Leviathan, the true identity of the Hobbesgoblins' Lord-Over-All. After explaining his theory of government ("Let's Hear It For Leviathan", lyrics only) Hobbes and the monster attack for the game's first boss battle. The fight is immediately plagued by mishaps, including one incident where Phaidros's "Calvin's Predestined Hellfire" spell causes Hobbes to briefly turn into a Dire Tiger. When one of Leviathan's tentacles grab Cerune, she manifests a battle-axe of magic fire called the Axe of Separation and hacks the creature's arm off. She refuses to explain this power, but inspired by the small victory the party defeat Hobbes and reduce Leviathan into a state of Cartesian doubt; the confused monster vanishes into the depths, and the adventurers hurry to the other side and out of the Slough.

CHAPTER THREE: THE SHADOW CITY

Although our heroes make good time, they soon spot a detachment of Hobbesgoblins pursuing them. Afraid the goblins will be angry at the defeat of their god, the party hides; this turns out to be unnecessary, as the goblins only want Ephraim - the one who actually dealt the final blow against Leviathan - to be their new Lord-Over-All. Ephraim rejects the positions, and the party responds to the goblins' desperate pleading by suggesting a few pointers for creating a new society - punishing violence, promoting stability, reinforcing social behavior. The Hobbesgoblins grumble, but eventually depart - just in time for the party to be attacked by more of Buridan's Assassins. These killers' PLXM coins seem to suggest an origin in Xar-Morgoloth, the Shadow City, and indeed its jet-black walls now loom before them. But the city sits upon the only pass through the Central Mountains, so the party reluctantly enters.

Xar-Morgoloth turns out to be a pleasant town of white-washed fences and laughing children. In search of an explanation for the incongruity the five seek out the town's spiritual leader, the Priest of Lies. The Priest explains that although Xar-Morgoloth is superficially a nice place, the town is evil by definition. He argues that all moral explanations must be grounded in base moral facts that cannot be explained, whether these be respect for others, preference of pleasure over pain, or simple convictions that murder and theft are wrong. One of these base level moral facts, he says, is that Xar-Morgoloth is evil. It is so evil, in fact, that it is a moral imperative to keep people out of the city - which is why he sent assassins to scare them off.

Doubtful, the party seeks the mysterious visiting philosopher whom the Priest claimed originated these ideas: they find Immanuel Kant living alone on the outskirts of the city. Kant tells his story: he came from a parallel universe, but one day a glowing portal appeared in the sky, flinging him into the caves beyond Xar-Morgoloth. Wandering into Xar-Morgoloth, he tried to convince the citizens of his meta-ethical theories, but they insisted they could ground good and evil in basic moral intuitions instead. Kant proposed that Xar-Morgoloth was evil as a thought experiment to disprove them, but it got out of hand.

When our heroes challenge Kant's story and blame him for the current state of the city, Kant gets angry and casts Parmenides' Stasis Hex, freezing them in place. Then he announces his intention to torture and kill them all. For although in this world Immanuel Kant is a moral philosopher, in his own world (he explains) Immanuel Kant is a legendary villain and figure of depravity ("I'm Evil Immanuel Kant", lyrics only). Cerune manifests a second magic weapon, the Axe of Choice, to break the Stasis Hex, and the party have their second boss battle, which ends in defeat for Evil Kant. Searching his home, they find an enchanted Parchment of Natural Law that causes the chill in the air whenever the city's name is spoken.

Armed with this evidence, they return to the Priest of Lies and convince him that his moral theory is flawed. The Priest dispels the shadow over the city, recalls his assassins, and restores the town name to its previous non-evil transliteration of Summerglass. He then offers free passage through the caverns that form the only route through the Central Mountains.

CHAPTER FOUR: THE CAVERNS OF ABCISSA

Inside the caverns, which are nearly flooded by the overflowing Abcissa River, the party encounter an army of Water Elementals, leading them to suspect that they may be nearing the headquarters of Heraclitus' arch-enemy, Thales. The Water Elementals are mostly busy mining the rock for gems and magic artifacts, but one of them is sufficiently spooked by Phaidros to cast a spell on him, temporarily turning him to water. This is not immediately a disaster - Phaidros assumes a new form as a water elemental but keeps his essential personality - except that in an Ephraimesque display of overexperimention, Phaidros wonders what would happen if he temporarily relaxed the morphogenic field that holds him in place - as a result, he loses his left hand, a wound which stays in place when he reverts back to his normal form a few hours later. A resigned Phaidros only quotes the Bible: ("And if your hand offend you, cut it off: it is better for you to enter into life maimed, than having two hands to go into hell" - Mark 9:43) and trusts in the Divine plan.

The Caverns of Abcissa are labyrinthine and winding, but eventually the party encounters a trio who will reappear several times in their journey: Ruth (who tells the truth), Guy (who'll always lie) and Clancy (who acts on fancy). These three have a habit of hanging around branching caverns and forks in the road, and Ephraim solves their puzzle thoroughly enough to determine what route to take to the center of the cave system.

Here, in a great cavern, lives a civilization of cave-men whose story sounds a lot like Evil Kant's - from another world, minding their own business until a glowing portal appeared in the sky and sucked them into the caves. The cave-men are currently on the brink of civil war after one of their number, Thag, claims to have visited the mythical "outside" and discovered a world of magic and beauty far more real than the shadows dancing on the walls of their cavern. Most of the other cave-men, led by the very practical Vur, have rejected his tale, saying that the true magic and beauty lies in accepting the real, in-cave world rather than chasing after some outside paradise - but a few of the youth have flocked to Thag's banner, including Antil, a girl with mysterious magic powers.

Only the timely arrival of the adventurers averts a civil war; the party negotiates a truce and offers to solve the dispute empirically - they will escort Vur and Antil with them through the caverns so that representatives of both sides can see whether or not the "outside" really exists. This calms most of the cave-men down, and with Vur and Antil alongside, they head onward to the underground source of the Abcissa - which, according to their research, is the nerve center of Thales' watery empire.

On the way, they encounter several dangers. First, they awake a family of hibernating bears, who are quickly dispatched but who manage to maul the frail Vur so severely that only some divine intervention mediated by Phaidros saves his life. Second, they come across a series of dimensional portals clearly linked to the stories related by Evil Kant and the cave-men. Some link directly to otherworldly seas, pouring their water into the Abcissa and causing the recent floods. Others lead to otherworldly mines and quarries, and are being worked by gangs of Water Elementals. After some discussion of the ethics of stranding the Water Elementals, the five philosophers decide to shut down as many of the portals as possible.

They finally reach the source of the Abcissa, and expecting a battle, deck themselves out in magic armor that grants immunity to water magic. As expected, they encounter Thales, who reveals the full scale of his dastardly plot - to turn the entire world into water. But his exposition is marred by a series of incongruities, including his repeated mispronunciations of his own name ("All is Water", lyrics only). And when the battle finally begins, the party dispatches Thales with minimal difficulty, and the resulting corpse is not that of a Greek philosopher at all, but rather that of Davidson's Swampman, a Metaphysical summon that can take the form of any creature it encounters and imitate them perfectly.

Before anyone has time to consider the implications of their discovery, they are attacked by the real Water Mage, who bombards them with powerful water spells to which their magic armor mysteriously offers no protection. Worse, the Mage is able to create dimensional portals at will, escaping attacks effortlessly. After getting battered by a series of magic Tsunamis that nearly kill several of the weaker party members, the adventurers are in dire straits.

Then the tide begins to turn. Antil manifests the power to go invisible and attack the Water Mage from an unexpected vantage. Cerune manifests another magic weapon, the Axe of Extension, which gives her allies the same powers over space as the Water Mage seems to possess. And with a little prompting from Cerune, Phaidros and Nomophilos realize the Water Mage's true identity. Magic armor doesn't grant protection from his water spells because they are not water at all, but XYZ, a substance physically identical to but chemically different from H2O. And his mastery of dimensional portals arises from his own origin in a different dimension, Twin Earth. He is Hilary Putnam ("All is Water, Reprise", lyrics only) who has crossed dimensions, defeated Thales, and assumed his identity in order to take over his watery empire and complete his world domination plot. With a last push of magic, the party manage to defeat Putnam, who is knocked into the raging Abcissa and drowned in the very element he sought to control.

They tie up the loose ends of the chapter by evacuating the Water Elementals from Twin Earth, leading the cave-men to the promised land of the Outside, and confronting Antil about her mysterious magic. Antil gives them the source of her power to turn invisible: the Ring of Gyges, which she found on the cave floor after an earthquake. She warns them never to use it, as it presents a temptation which their ethics might be unable to overcome.

CHAPTER FIVE: CLIMBING MOUNT IMPROBABLE

Now back on the surface, the party finds their way blocked by the towering Mount Improbable, which at first seems too tall to ever climb. But after some exploration, they find there is a gradual path sloping upward, and begin their ascent. They are blocked, however, by a regiment of uniformed apes: cuteness turns to fear when they get closer and find the apes have machine guns. They decide to negotiate, and the apes prove willing to escort them to their fortress atop the peak if they can prove their worth by answering a few questions about their religious beliefs.

Satisfied, the ape army lead them to a great castle at the top of the mountain where Richard Dawkins ("Beware the Believers", credit Michael Edmondson) and his snow leopard daemon plot their war against the gods themselves. Dawkins believes the gods to be instantiated memes - creations of human belief that have taken on a life of their own due to Aleithos' madness - and accuses them of causing disasters, poverty, and ignorance in order to increase humanity's dependence upon them and keep the belief that sustains their existence intact. With the help of his genetically engineered apes and a fleet of flying battleships, he has been waging war against all the major pantheons of polytheism simultaneously. Dawkins is now gearing up to attack his most implacable foe, Jehovah Himself, although he admits He has so far managed to elude him.

Hoping the adventurers will join his forces, he takes them on a tour of the castle, showing them the towering battlements, the flotilla of flying battleships, and finally, the dungeons. In these last are imprisoned Fujin, Japanese god of storms; Meretseger, Egyptian goddess of the flood, and even Ares, the Greek god of war (whom Dawkins intends to try for war crimes: not any specific war crime, just war crimes in general). When the party reject Dawkins' offer to join his forces (most vocally Phaidros, most reluctantly Ephraim) Dawkins locks them in the dungeons themselves.

They are rescued late at night by their old friend Theseus. Theseus lost his ship in a storm (caused by the Japanese storm god, Fujin) and joined Dawkins' forces to get revenge; he is now captain of the aerial battleships. Theseus loads the adventurers onto a flying battleship and deposits them on the far side of the mountain, where Dawkins and his apes will be unlikely to find them.

Their troubles are not yet over, however, for they quickly encounter a three man crusade consisting of Blaise Pascal, Johann Tetzel, and St. Augustine of Hippo (mounted, cavalry-style, upon an actual hippopotamus). The three have come, led by a divine vision, to destroy Dawkins and his simian armies as an abomination unto the Lord, and upon hearing that the adventurers have themselves escaped Dawkins, invite them to come along. But the five, despite their appreciation for Pascal's expository fiddle music ("The Devil and Blaise Pascal") are turned off by Tetzel's repeated attempts to sell them indulgences, and Augustine's bombastic preaching. After Phaidros gets in a heated debate with Augustine over the role of pacifism in Christian thinking, the two parties decide to go their separate ways, despite Augustine's fiery condemnations and Pascal's warning that there is a non-zero chance the adventurers' choice will doom them to Hell.

After another encounter with Ruth, Guy, and Clancy, our heroes reach the base of Mount Improbable and at last find themselves in Russellia.

CHAPTER SIX: THE PALL OVER RUSSELLIA

Russellia is, as the legends say, shrouded in constant darkness. The gloom and the shock of being back in her ancestral homeland are too much for Cerune, who breaks down and reveals her last few secrets. Before beginning the quest, she consulted the Turing Oracle in Cyberia, who told her to seek the aid of a local wizard, Zermelo the Magnificent. Zermelo gave her nine magic axes of holy fire, which he said possessed the power to break the curse over Russellia. But in desperation, she has already used three of the magic axes, and with only six left she is uncertain whether she will have the magic needed.

At that moment, Heraclitus appears in a burst of flame, seeking a debriefing on the death of his old enemy Thales. After recounting the events of the past few weeks, our heroes ask Heraclitus whether, as a Fire Mage, he can reforge the axes of holy fire. Heraclitus admits the possibility, but says he would need to know more about the axes, their true purpose, and the enemy they were meant to fight. He gives the party an enchanted matchbook, telling them to summon him by striking a match when they gather the information he needs.

Things continue going wrong when, in the midst of a discussion about large numbers, Phaidros makes a self-contradictory statement that summons a Paradox Beast. Our heroes stand their ground and manage to destroy the abomination, despite its habit of summoning more Paradox Beasts to its aid through its Principle of Explosion spell. Bruised and battered, they limp into the nearest Russellian city on their map, the town of Ravenscroft.

The people of Ravenscroft tell their story: in addition to the eternal darkness, Russellia is plagued by vampire attacks and by a zombie apocalypse, which has turned the population of the entire country, save Ravenscroft, into ravenous brain-eating zombies. Despite the burghers claiming the zombie apocalypse had been confirmed by no less a figure than Thomas Nagel, who passed through the area a century ago, our heroes are unconvinced: for one thing, the Ravenscrofters are unable to present any evidence that the other Russellians are zombies except for their frequent attacks on Ravenscroft - and the Ravenscrofters themselves attack the other towns as a "pre-emptive measure". But the Ravenscrofters remain convinced, and even boast of their plan to launch a surprise attack on neighboring Brixton the next day.

Suspicious, our heroes head to the encampment of the Ravenscroft army, where they are just in time to see Commander David Chalmers give a rousing oration against the zombie menace ("Flee! A History of Zombieism In Western Thought", credit Emerald Rain). They decide to latch on to Chalmers' army, both because it is heading the same direction they are and because they hope they may be able to resolve the conflict between Ravenscroft and Brixton before it turns violent.

They camp with the army in some crumbling ruins from the golden age of the Russellian Empire. Entering a ruined temple, they disarm a series of traps to enter a vault containing a legendary artifact, the Morningstar of Frege. They also encounter a series of statues and bas-reliefs of the Good King, in which he demonstrates his chivalry by swearing an oath to Aleithos that he will defend all those who cannot defend themselves. Before they can puzzle out the meaning of all they have seen, they are attacked by vampires, confirming the Ravenscrofters' tales; they manage to chase them away with their magic and a hare-brained idea of Phaidros' to bless their body water, turning it into holy water and burning them up from the inside.

The next morning, they sneak into Brixton before the main army, and find their fears confirmed: the Brixtonites are normal people, no different from the Russellians, and they claim that Thomas Nagel told them that they were the only survivors of the zombie apocalypse. They manage to forge a truce between Ravenscroft and Brixton, but to their annoyance, the two towns make peace only to attack a third town, Mountainside, which they claim is definitely populated by zombies this time. In fact, they say, the people of Mountainside openly admit to being zombies and don't even claim to have souls.

Once again, our heroes rush to beat the main army to Mountainside. There they find the town's leader, Daniel Dennett, who explains the theory of eliminative materialism ("The Zombies' Secret"). The party tries to explain the subtleties of Dennett's position to a bloodthirsty Chalmers, and finally all sides agree to drop loaded terms like "human" and "zombie" and replace them with a common word that suggests a fundamental humanity but without an internal Cartesian theater (one of our heroes suggests "NPC", and it sticks). The armies of the three towns agree to ally against their true common enemy - the vampires who live upon the Golden Mountain and kidnap their friends and families in their nighttime raids.

Before the attack, Nomophilos and Ephraim announce their intention to build an anti-vampire death ray. The theory is that places on the fringe of Russellia receive some sunlight, while places in the center are shrouded in endless darkness. If the towns of Russellia can set up a system of mirrors from their highest towers, they can reflect the sunlight from the borderlands into a central collecting mirror in Mountainside, which can be aimed at the vampires' hideout to flood it with daylight, turning them to ashes. Ephraim, who invested most of his skill points into techne, comes up with schematics for the mirror, and after constructing a successful prototype, Chalmers and Dennett sound the attack order.

The death ray takes out many of the vampires standing guard, but within their castle they are protected from its light: our heroes volunteer to infiltrate the stronghold, but are almost immediately captured and imprisoned - the vampires intend to sacrifice Cerune in a ritual to use her royal blood to increase their power. But the adventurers make a daring escape: arch-conservative Nomophilos uses the invisible hand of the marketplace to steal the keys out of the jailer's pocket, and Phaidros summons a five hundred pound carnivorous Christ metaphor to maul the guards. Before the party can escape the castle, they are confronted by the vampire lord himself, who is revealed to be none other than Thomas Nagel ("What Is It Like To Be A Bat?"). In the resulting battle, Nagel is turned to ashes and the three allied cities make short work of the remaining vampires, capturing the castle.

The next morning finds our heroes poring over the vampire lord's library. Inside, they find an enchanted copy of Godel Escher Bach (with the power to summon an identical enchanted copy of Godel Escher Bach) and a slew of books on Russellian history. Over discussion of these latter, they finally work out what curse has fallen over the land, and what role the magic axes play in its removal.

[spoiler alert; stop here if you want to figure it out for yourself]

The Good King's oath to defend those who could not defend themselves was actually more complicated than that: he swore an oath to the god Aleithos to defend those and only those who could not defend themselves. His enemies, realizing the inherent contradiction, attacked him, trapping Russell in a contradiction - if he defended himself, he was prohibited from doing so; if he did not defend himself, he was obligated to do so. Trapped, he was forced to break his oath, and the Mad God punished him by casting his empire into eternal darkness and himself into an endless sleep.

The nine axes of Zermelo the Magnificent embody the nine axioms of ZFC. If applied to the problem, they will allow set theory to be reformulated in a way that makes the paradox impossible, lifting the curse and waking the Good King.

Upon figuring out the mystery, the party strike the enchanted match and summon Heraclitus, who uses fire magic to reforge the Axes of Choice, Separation, and Extension. Thus armed, the party leave the Vampire Lord's castle and enter the system of caverns leading into the Golden Mountain.

CHAPTER SEVEN: THE KING UNDER THE MOUNTAIN


The party's travels through the cavern are quickly blocked by a chasm too deep to cross. Nomophilos saves the day by realizing that the enchanted copy of Godel Escher Bach creates the possibility of infinite recursion; he uses each copy of GEB to create another copy, and eventually fills the entire chasm with books, allowing the party to walk through to the other side.

There they meet Ruth, Clancy, and Guy one last time; the three are standing in front of a Logic Gate, and to open it the five philosophers must solve the Hardest Logic Puzzle Ever. In an epic feat that the bards will no doubt sing for years to come, Macx comes up with a solution to the puzzle, identifies each of the three successfully, and opens the Logic Gate.

Inside the gate is the Good King, still asleep after two centuries. His resting place is guarded by the monster he unleashed, a fallen archangel who has become a Queen Paradox Beast. The Queen summons a small army of Paradox Beast servants with Principle of Explosion, and the battle begins in earnest. Cerune stands in a corner, trying to manifest her nine magic axes, but Nomophilos uses his Conservative spell "Morning in America" to summon a Raygun capable of piercing the Queen Paradox Beast's armored exoskeleton. Macx summons a Universal Quantifier and attaches it to his Banish Paradox Beast spell to decimate the Queen's armies. Ephraim desperately tries to wake the Good King, while Phaidros simply prays.

After an intense battle, Cerune manifests all nine axes and casts them at the Queen Paradox Beast, dissolving the paradox and destroying the beast's magical defenses. The four others redouble their efforts, and finally manage to banish the Queen. When the Queen Paradox Beast is destroyed, Good King Bertrand awakens.

Bertrand is temporarily discombobulated, but eventually regains his bearings and listens to the entire adventure. Then he tells his story. The attack that triggered the curse upon him, he says, was no coincidence, but rather a plot by a sinister organization against whom he had been waging a shadow war: the Bayesian Conspiracy. He first encountered the conspiracy when their espionage arm, the Bayes Network, tried to steal a magic emerald of unknown origin from his treasury. Since then, he worked tirelessly to unravel the conspiracy, and had reached the verge of success - learning that their aim was in some way linked to a plan to gain the shattered power of the Mad God Aleithos for themselves - when the Conspiracy took advantage of his oath and managed to put him out of action permanently.

He is horrified to hear that two centuries have passed, and worries that the Bayesians' mysterious plan may be close to fruition. He begs the party to help him re-establish contact with the Conspiracy and continue figuring out their plans, which may be a dire peril to the entire world. But he expresses doubt that such a thing is even possible at this stage.

In a burst of flame, Heraclitus appears, announcing that all is struggle and that he has come to join in theirs. He admits that the situation is grim, but declares it is not as hopeless as it seems, because they do not fight alone. He invokes the entire Western canon as the inspiration they follow and the giants upon whose shoulders they stand ("Grand Finale").

Heraclitus, Good King Bertrand, and the five scholars end the adventure by agreeing to seek out the Bayesian Conspiracy and discover whether Russell's old adversaries are still active. There are nebulous plans to continue the campaign (subject to logistical issues) in a second adventure, Fermat's Last Stand.

MUSIC

LYRICS ONLY
Hobbes' Song: Let's Hear It For Leviathan
Kant's Song: I'm Evil Immanuel Kant
Thales' Song: All Is Water
Putnam's Song: All Is Water, Reprise

GOOD ARTISTS BORROW, GREAT ARTISTS STEAL
Dawkins' Song: Beware The Believers (credit: Michael Edmondson)
Chalmers' Song: Flee: A History of Zombieism In Western Thought (credit: Emerald Rain)

ORIGINAL ADAPTATIONS
Pascal's Song: The Devil and Blaise Pascal
Dennett's Song: The Zombies' Secret
Vampire Nagel's Song: What Is It Like To Be A Bat?
Heraclitus' Song: Grand Finale

Comment author: Yvain 12 November 2011 12:46:26PM 0 points [-]

Anyone want to make a guess with confidence intervals how many people have taken the survey at this point? (noon GMT on 12 November)

Comment author: MixedNuts 11 November 2011 05:23:34PM 2 points [-]

90% of people who used it in the context you did would be using it seriously

Uh. Okay. I guess I far underestimated the proportion of people who would seriously call you a bad person on LW. My bad.

Comment author: Yvain 12 November 2011 02:35:37AM 0 points [-]

Oh, fine. Maybe I'm just oversensitive. Downvote revoked.

Comment author: MixedNuts 11 November 2011 04:39:39PM 0 points [-]

No, just the description that is intended to make people go "Ew, undateable" (obesity, poor hygiene), as opposed to "Aw, poor girl, those guys are so shallow" (ugly duckling).

But... but... how come I don't get to say that, when you get to say "This is a terrible debate and you should all feel bad for having it."? (Because you're freaking Yvain. Also because you have some concept of tact.)

Comment author: Yvain 11 November 2011 05:07:36PM 8 points [-]

Again, where did I say that it was "gross"?

I said it would make it harder for the woman to get dates with men, but is that really in doubt? Do you need me to find statistics showing that (American) men in general rate women who don't shave their legs as less attractive? And I was using it as an example of something that shouldn't matter, but does.

You don't get to say that because 90% of people who used it in the context you did would be using it seriously, and because accusing someone of being a bad person for being sexist is more of a trigger point than accusing someone of having a bad debate.

Comment author: MixedNuts 11 November 2011 07:07:28AM *  0 points [-]

The obvious strategy for this woman seems to be to look specifically for men who don't care much about looks and hygiene. (Also, you're a bad person for saying a woman who doesn't shave her legs is gross.) Melissa McEwan is fat and doesn't shave her legs (though as far as I know she has good hygiene), and that works out just fine because the people she's interested in prefer, or at least don't strongly disprefer, that.

On the other hand, those compassionate betas (at least those we hear complaining) seem to only pursue the (admittedly common) type of women who care strongly about status. There are obvious reasons for that (it correlates with being conventionally attractive), but it does seem like they're shooting themselves in the foot. If people who prefer your type have to throw themselves at you before you notice them, you're doing it wrong.

Edit: I don't understand the downvotes. wedrifid's objection is true, but it wasn't my main point. Is it because I'm telling people to hit on people who aren't their first choice? Or is it the "how dare you want the same characteristics everyone wants" undertones? Or did I just plain miss Yvain's point?

Comment author: Yvain 11 November 2011 04:28:45PM *  9 points [-]

I voted you down for saying "Also, you're a bad person for saying a woman who doesn't shave her legs is gross" when I never said anything of the sort. Maybe you misunderstood the term "grossly obese" (which uses 'gross' in the sense of 'large')? I don't know.

Even if I had said that, there would have to be a nicer way to correct it.

Comment author: usedToPost 08 November 2011 07:11:39PM *  1 point [-]

Luke originally tried to write an article referring to PUA. People told him this was controversial

Yes, I would also like to congratulate Lukeprog for caving in to social pressure and posting information which is deliberately misleading. I am sure that all the (male) people who read this article, and start using his politically correct nonsense to improve their dating lives will really appreciate it too! (As for female dating advice, I don't know what I am talking about, so I will shut up)

Since the advice given in the article is actively harmful, a better solution would be for Lukeprog to just tell people to google pick up. That way, nobody could flame him on LW, and he wouldn't be spreading actively harmful information.

Now he seems to be writing more or less the same thing, but communicating it in a less offensive way

he is telling people to display "agreeableness" - pretty much the opposite of PUA advice, he is telling you to "like" others - a dangerous piece of advice which could quickly turn into desperate, supplicative behavior, complimenting, etc. He is emphasizing physical looks over dominance and alpha-male behavior, again the opposite of PUA advice.

I will edit my comment to take account of what you said.

Comment author: Yvain 08 November 2011 07:29:08PM *  12 points [-]

Your point about agreeableness is well taken, so I looked up his reference, Figueredo et al. (2006).

First, keep in mind he's using agreeableness in the OCEAN sense, not in the sense of "a person who always agrees to everything". So it's not diametrically opposed to PUA belief, although I agree there's still a problem that has to be explained.

That brings us to the reference. Figueredo's study itself found no impact of agreeableness, but in the introduction, it cites eight previous studies that it said found "extraversion, openness, and agreeableness are reliably correlated with mating success". I looked up one of these studies, and it was on the success of long-term marital relationships, which is a whole different kettle of fish than the PUA's usual focus. So depending on the other seven studies I didn't have the energy to look up, they could both be right. It would have been nice if Luke had qualified that in his post, but really the fault was on Figueredo and not him.

Other than that, I would honestly like to hear what advice of Luke's you consider misleading. Again aside from the "Mean and Variance" section, it all seems pretty well referenced and backed up.

Comment author: Yvain 03 November 2011 11:22:05PM *  109 points [-]

This is a terrible debate and you should all feel bad for having it. Now let me join in.

The research on this topic is split into "completely useless" and "mostly useless". In the former category we have studies that, with a straight face, purport to show that women like nice guys by asking women to self-report on their preferences. To illuminate just how silly this is, consider the mirror case of asking men "So, do you like witty charming girls with good personalities, or supermodels with big breasts?" When this was actually done, men rated "physical attractiveness" only their 22nd most important criterion for a mate - number one was "sincerity", and number nineteen was "good manners". And yet there are no websites where you can spend $9.95 per month to stream videos of well-mannered girls asking men to please pass the salad fork, and there are no spinster apartments full of broken-hearted supermodels who just didn't have enough sincerity. So self-reports are right out.

Other-reports may be slightly less silly. Herold and Milhausen, 1999, found that 56% of university women believed that women in general were more likely to date jerks than nice guys. But although women may have less emotional investment in the issue than men, their opinions are still just opinions.

The few studies that earn the coveted accolade of "only mostly useless" are those that try to analyze actual behavior. Bogart and Fisher typify a group of studies that show that good predictors of a man's number of sexual partners include disinhibitedness, high testosterone levels, "hypermasculinity", "sensation seeking", antisocial personality, and extraversion. Meston et al typify a separate group of studies on sex and the Big Five traits when she says that "agreeableness was the most consistent predictor of behavior...disagreeable men and women were more likely to have had sexual intercourse and with a greater number of partners than agreeable men and women. Nonvirgins of both sexes were more likely to be calculating, stubborn, and arrogant in their interpersonal behavior than virgins. Neuroticism predicted sexual experience in males only; timid, unassertive men were less sexually experienced than emotionally stable men...the above findings were all statistically significant at p<.01"

These studies certainly show that jerkishness is associated with high number of sexual partners, but they're not quite a victory for the "nice guys finish last" camp for a couple of reasons. First, men seem to come off almost as bad as women do. Second, there's no reason to think that any particular "nice" woman will like jerks; many of the findings could be explained by disagreeable men hooking up with disagreeable women, disagreeing with them about things (as they do) and then breaking up and hooking up with other disagreeable women, while the agreeable people form stable pair bonds. Boom - disagreeable people showing more sexual partners than agreeable people.

I find more interesting the literature about intelligence and sexual partners. In high-schoolers, each extra IQ point increases chance of virginity by 2.7% for males and 1.7% by females. 87% of 19-year old US college students have had sex, yet only 65% of MIT graduate students have had sex. There's conflicting research about whether this reflects lower sex drive in these people or less sexual success; it's probably a combination of both. See linked article for more information.

The basic summary of the research seems to be that smart, agreeable people complaining that they have less sex than their stupid, disagreeable counterparts are probably right, and that this phenomenon occurs both in men and women but is a little more common in men.

Moving from research to my own observations, I do think there are a lot of really kind, decent, shy, nerdy men who can't find anyone who will love them because they radiate submissiveness and non-assertiveness, and women don't find this attractive. Most women do find dominant, high-testosterone people attractive, and dominance and testosterone are risk factors for jerkishness, but not at all the same thing and women can't be blamed for liking people with these admittedly attractive characteristics.

There are also a lot of really kind, decent, shy, nerdy women who can't find anyone who will love them because they're not very pretty. Men can't be blamed for liking people they find attractive either, but this is also sad.

But although these two situations are both sad, at the risk of being preachy I will say one thing. When a girl is charming and kind but not so conventionally attractive, and men avoid her, and this makes her sad...well, imagine telling her that only ugly people would think that, and since she's ugly she doesn't deserve a man, and she probably just wants to use him for his money anyway because of course ugly women can't genuinely want love in the same way anyone else would (...that would be unfair!) This would be somewhere between bullying and full on emotional abuse, the sort of thing that would earn you a special place in Hell.

Whereas when men make the same complaint, that they are nice and compassionate but not so good at projecting dominance, there is a very large contingent of people, getting quite a lot of respect and validation from the parts of society that should know better, who immediately leap out to do their best to make them feel miserable - to tell that they don't deserve a relationship, that they're probably creeps who are only in it for the sex and that if they were a real man they'd stop whining about being "entitled to sex".

EDIT: But see qualification here

Comment author: Yvain 08 November 2011 07:07:36PM *  24 points [-]

After talking to a couple of people about this, I should qualify/partially-retract the original comment.

Some people have suggested to me that the best metaphor a man can use to understand how women think about "nice guys" isn't an ugly duckling woman who gets turned down by the men she likes, but a grossly obese woman who never showers or shaves her legs, and who goes around complaining loudly to everyone she knows that men are all vapid pigs who are only interested in looks.

I would find this person annoying, and although I hope I would be kind enough not to lash out against her in quite the terms I mentioned above, I would understand the motivations of someone who did, instead of having to classify him as having some sort of weird Martian brain design that makes him a moral monster.

The obesity metaphor is especially relevant. Since there are people out there who think becoming skinny is as easy as "just eat less food", I can imagine people who think becoming socially assertive really is as easy as "just talk to people and be more confident".

For people who honestly believe those things, and there seem to be a lot of them, the obese woman and the socially awkward man would reduce to the case of the woman who never showered but constantly complained about how superficial men were to reject her over her smell - annoying and without any redeeming value.

Comment author: usedToPost 08 November 2011 03:51:30PM *  6 points [-]

Lukeprog, you have produced exactly that which we have been warned against: an article and a paradigm which has all the appearances and dressings of rationality (lots of citations, links to articles on decision theory, rationalist lingo), but which spectacularly fails to actually pursue the truth.

Vladimir_M puts it better than I could:

First, there is the conspicuous omission of any references to the PUA elephant in the room. The body of insight developed by this particular sort of people, whatever its faults, is of supreme practical importance for anyone who wants to formulate practical advice in this area. Without referencing it explicitly, one can either ignore it altogether and thus inevitably talk nonsense, or pretend to speak based solely on official academic literature, which is disingenuous and unfair in its failure to attribute credit and also misleading for those who would like to pursue their own research in the matter.....

he continues:

On the whole, the article is based on the premise that an accurate and no-nonsense analysis of the topic will result in something that sounds not just inoffensive, but actually strongly in line with various fashionable and high-status norms and ideals of the broader society. This premise however is flawed, and those who believe that this has in fact been accomplished should apply the powerful debiasing heuristic that says that when a seemingly rational discussion of some deeply problematic and controversial topic sounds pleasant and reassuring, there's probably something fishy going on

And finally:

So, what about the quality of advice that will be produced by a LW discussion on these topics operating under such constraints of respectability, where disreputable sources of accurate information are tabooed, a pretense must be maintained that the discourse is grounded in officially accredited scholarship and other high-status sources of information, and -- most important of all -- the entire discourse and its bottom line must produce a narrative that is in line with the respectable, high-status views of humanity and society? I am not at all optimistic, especially having seen what has been produced so far!

Yvain is also on point:

shy, nerdy men who can't find anyone who will love them because they radiate submissiveness and non-assertiveness, and women don't find this attractive. Most women do find dominant, high-testosterone people attractive

In three worlds collide, we were introduced to the "Order of Silent Confessors", which is "charged with guarding sanity, not morality". In this post especially, I feel that sanity is lying beaten and abused on the floor. I think we need the "Order of Silent Confessors" now.

As a start, Lukeprog, I think you should include the exerpts by vladimir_M and Yvain above in your article.

Comment author: Yvain 08 November 2011 06:50:54PM 11 points [-]

Thank you for the positive mention, but I'm afraid I disagree with your model of me. Luke is a far braver man than I to even enter this minefield; I won't condemn him for not dancing a merry jig on top of it too.

Luke originally tried to write an article referring to PUA. People told him this was controversial, not just among ignorant people but among long-time readers of this site, that it had always led to unpleasant flame wars in the past, and that it was making us look bad "abroad".

Now he seems to be writing more or less the same thing, but communicating it in a less offensive way. I don't fault him for leaving anything out yet because it's only been one post in a series. I don't think anything he wrote is actually false (well, I have issues with the 'Mean and Variance' section, but he retracted the meat of that). And I think he made the right decision in trying to pitch it to a wider audience.

Comment author: Username 06 November 2011 07:16:20PM *  10 points [-]

Yvain, one very important question that I think you missed: Do you currently have an account on Lesswrong?

I personally don't, and glancing through the number of 'first post' comments here, I believe that the ratio of lurkers to active users may be significant. (This is a throwaway account, and I am making an exception this once because there would be no other way to get information from the lurkers.)

Comment author: Yvain 06 November 2011 07:19:14PM 6 points [-]

Good point. I hope that the "karma" question will take care of some of the problem, but I should have distinguished more finely.

Comment author: NancyLebovitz 03 November 2011 04:57:41AM 22 points [-]

Why do girls say they want "nice guys" but date only "jerks"?

I find that claim bewildering because the partnered men I know aren't jerks. It could be that I'm filtering for non-jerkness, but my tentative alternate theory is that the maybe the most conspicuously attractive women prefer jerks, and the men who resent the pattern aren't noticing most women. Or possibly a preference for jerks really is common in "girls"-- not children, but women below some level of maturity (age 25? 30? whatever it takes to get tired of being mistreated?), and some men are imprinted on what they saw in high school.

For those of you who believe that women prefer jerks, what sort of behavior do you actually mean? What proportion of women are you talking about? Is there academic research to back this up? What have you seen in your social circle?

Comment author: Yvain 03 November 2011 11:22:05PM *  109 points [-]

This is a terrible debate and you should all feel bad for having it. Now let me join in.

The research on this topic is split into "completely useless" and "mostly useless". In the former category we have studies that, with a straight face, purport to show that women like nice guys by asking women to self-report on their preferences. To illuminate just how silly this is, consider the mirror case of asking men "So, do you like witty charming girls with good personalities, or supermodels with big breasts?" When this was actually done, men rated "physical attractiveness" only their 22nd most important criterion for a mate - number one was "sincerity", and number nineteen was "good manners". And yet there are no websites where you can spend $9.95 per month to stream videos of well-mannered girls asking men to please pass the salad fork, and there are no spinster apartments full of broken-hearted supermodels who just didn't have enough sincerity. So self-reports are right out.

Other-reports may be slightly less silly. Herold and Milhausen, 1999, found that 56% of university women believed that women in general were more likely to date jerks than nice guys. But although women may have less emotional investment in the issue than men, their opinions are still just opinions.

The few studies that earn the coveted accolade of "only mostly useless" are those that try to analyze actual behavior. Bogart and Fisher typify a group of studies that show that good predictors of a man's number of sexual partners include disinhibitedness, high testosterone levels, "hypermasculinity", "sensation seeking", antisocial personality, and extraversion. Meston et al typify a separate group of studies on sex and the Big Five traits when she says that "agreeableness was the most consistent predictor of behavior...disagreeable men and women were more likely to have had sexual intercourse and with a greater number of partners than agreeable men and women. Nonvirgins of both sexes were more likely to be calculating, stubborn, and arrogant in their interpersonal behavior than virgins. Neuroticism predicted sexual experience in males only; timid, unassertive men were less sexually experienced than emotionally stable men...the above findings were all statistically significant at p<.01"

These studies certainly show that jerkishness is associated with high number of sexual partners, but they're not quite a victory for the "nice guys finish last" camp for a couple of reasons. First, men seem to come off almost as bad as women do. Second, there's no reason to think that any particular "nice" woman will like jerks; many of the findings could be explained by disagreeable men hooking up with disagreeable women, disagreeing with them about things (as they do) and then breaking up and hooking up with other disagreeable women, while the agreeable people form stable pair bonds. Boom - disagreeable people showing more sexual partners than agreeable people.

I find more interesting the literature about intelligence and sexual partners. In high-schoolers, each extra IQ point increases chance of virginity by 2.7% for males and 1.7% by females. 87% of 19-year old US college students have had sex, yet only 65% of MIT graduate students have had sex. There's conflicting research about whether this reflects lower sex drive in these people or less sexual success; it's probably a combination of both. See linked article for more information.

The basic summary of the research seems to be that smart, agreeable people complaining that they have less sex than their stupid, disagreeable counterparts are probably right, and that this phenomenon occurs both in men and women but is a little more common in men.

Moving from research to my own observations, I do think there are a lot of really kind, decent, shy, nerdy men who can't find anyone who will love them because they radiate submissiveness and non-assertiveness, and women don't find this attractive. Most women do find dominant, high-testosterone people attractive, and dominance and testosterone are risk factors for jerkishness, but not at all the same thing and women can't be blamed for liking people with these admittedly attractive characteristics.

There are also a lot of really kind, decent, shy, nerdy women who can't find anyone who will love them because they're not very pretty. Men can't be blamed for liking people they find attractive either, but this is also sad.

But although these two situations are both sad, at the risk of being preachy I will say one thing. When a girl is charming and kind but not so conventionally attractive, and men avoid her, and this makes her sad...well, imagine telling her that only ugly people would think that, and since she's ugly she doesn't deserve a man, and she probably just wants to use him for his money anyway because of course ugly women can't genuinely want love in the same way anyone else would (...that would be unfair!) This would be somewhere between bullying and full on emotional abuse, the sort of thing that would earn you a special place in Hell.

Whereas when men make the same complaint, that they are nice and compassionate but not so good at projecting dominance, there is a very large contingent of people, getting quite a lot of respect and validation from the parts of society that should know better, who immediately leap out to do their best to make them feel miserable - to tell that they don't deserve a relationship, that they're probably creeps who are only in it for the sex and that if they were a real man they'd stop whining about being "entitled to sex".

EDIT: But see qualification here

Comment author: thomblake 03 November 2011 03:53:55PM 3 points [-]

Seriously, dude, coding. Surely someone would be willing to volunteer to code a couple hundred open-ends. It should take like 5 minutes if you're willing to use broad brushstrokes. And if most of the raw data is made public, the later sifting for interesting tidbits is crowdsourced.

Comment author: Yvain 03 November 2011 07:43:14PM 2 points [-]

Well, sure, you could do that. But if I decided to hand-code all of the political write-ins into standard political terms like "liberal", "conservative", "etc", then all I'd end up with is a list of people's political preferences in a few bins of standard political terms.

Which is exactly what I have now when I don't allow write-ins. This way is easier for me and allows people to choose their bin themselves rather than have me try to guess whether some complicated philosophy is more conservative than libertarian or vice versa.

Comment author: [deleted] 03 November 2011 08:49:27AM *  9 points [-]

I took the survey.

The political section is begging for a one line write in, seriously. Please consider adding on in addition to the pick one option poll. I'm not having warm fuzzies for any of the groups and had to bite my tongue and pick one I really really dislike, just because the alternatives are so much worse and one of the alternatives, while probably quite popular a choice, will be misinterpreted if I chose it.

In response to comment by [deleted] on 2011 Less Wrong Census / Survey
Comment author: Yvain 03 November 2011 01:38:23PM *  14 points [-]

From your perspective, that makes sense. From my perspective - I don't intend to ever look at this data. I'm going to import it into SPSS, have it crunch numbers for me, and come out with some result like "Less Wrong users are 65% libertarian" or like "Men are more likely to be socialist than women."

If you put "other" - and this applies to any of the questions, not just this one - you're pretty much wasting your vote unless someone else is going to sift through the data and be interested that this particular anonymous line of the spreadsheet believes in strong environmental protection but an otherwise free market.

Looking at the answers, I really shouldn't have allowed write-ins for any questions - I was kind of surprised how many people can't settle on a specific gender, even though the aim of the question was more to figure out how many men versus women are on here than to judge how people feel about society (I considered saying "sex" instead, but that has its own pitfalls and wouldn't have let me get the transgender info as easily. I'll do it that way next time.)

I was particularly harsh on the politics question because I know how strong the temptation is. I think next survey I'll give every question an "other" check box, but it will literally just be a check box and there will be no room to write anything in.

Comment author: Oscar_Cunningham 02 November 2011 06:29:55PM 9 points [-]

When writing a comment on LessWrong, I often know exactly which criticisms people will give. I will have thought through those criticisms and checked that they're not valid, but I won't be able to answer them all in my post, because that would make my post so long that no-one would read it. It seems like I've got to let people criticise me, and then shoot them down. This seems awfully inefficient, it's like the purpose of having a discussion rather than me simply writing a long post is just to trick people into reading it.

Comment author: Yvain 02 November 2011 06:46:55PM 9 points [-]

Footnotes.

Comment author: Yvain 01 November 2011 08:20:23PM *  9 points [-]

Ohhhhhhh.... (quietly deletes previous comment)

2011 Less Wrong Census / Survey

77 Yvain 01 November 2011 06:28PM

The final straw was noticing a comment referring to "the most recent survey I know of" and realizing it was from May 2009. I think it is well past time for another survey, so here is one now.

Click here to take the survey

I've tried to keep the structure of the last survey intact so it will be easy to compare results and see changes over time, but there were a few problems with the last survey that required changes, and a few questions from the last survey that just didn't apply as much anymore (how many people have strong feelings on Three Worlds Collide these days?)

Please try to give serious answers that are easy to process by computer (see the introduction). And please let me know as soon as possible if there are any security problems (people other than me who can access the data) or any absolutely awful questions.

I will probably run the survey for about a month unless new people stop responding well before that. Like the last survey, I'll try to calculate some results myself and release the raw data (minus the people who want to keep theirs private) for anyone else who wants to examine it.

Like the last survey, if you take it and post that you took it here, I will upvote you, and I hope other people will upvote you too.

Comment author: saturn 01 November 2011 03:37:22AM 15 points [-]

I think it would be more informative to ask people to take one specific online test, now, and report their score. With everyone taking the same test, even if it's miscalibrated, people could at least see how they compare to other LWers. Asking people to remember a score they were given years ago is just going to produce a ridiculous amount of bias.

Comment author: Yvain 01 November 2011 08:39:48AM 6 points [-]

It's a bit late now, but if you recommend a particular test that's valid, short, and online, I can try that on the next survey.

Comment author: RichardKennaway 01 November 2011 08:00:52AM 4 points [-]

Why "Academics (on the teaching side)"? As an academic on the research side, what do I put?

Comment author: Yvain 01 November 2011 08:05:34AM 3 points [-]

Put "academic". I just meant "not a student"

Comment author: Yvain 01 November 2011 08:04:35AM 28 points [-]

After reading the feedback I've made the following changes (after the first 104 entries so that anyone who has access to the data can check if there are significant differences before and after these changes):

  • Added an "other" option in gender
  • Moved "date of singularity" above question mentioning 2100 to avoid anchoring. Really I should also move the Newton question for the same reason, but I'm not going to.
  • changed wording of anti-agathics question to "at least one person"
  • added a "don't know / no preference" to relationship style
  • clarified to answer probability as percent and not decimal; I'll go back and fix anyone who got this wrong, though. If you seriously mean a very low percent, like ".05%", please end with a percent mark so I know not to change it. Otherwise, leave the percent mark out.
  • Added a "government work" option.
  • Deleted "divorced". Divorced people can just put "single"
  • Added "economic/political collapse" to xrisk
  • Added "other" to xrisk
  • Added a question "Have you ever been to a Less Wrong meetup?" Please do NOT retake the survey to answer this question. I'll just grab statistics from the people who answered this after it was put up, while recognizing it might be flawed.

I did NOT add an "Other" to politics despite requests to do so, because I tried this last time and ended up with people sending me manifestos. I want to encourage people to choose whichever of those categories they're closest to. If you really don't identify at all with any of those categories, just leave it blank.

Comment author: NancyLebovitz 31 October 2011 09:08:30PM 4 points [-]

Speech patterns are part of culture, and you mentioned them as one of the things which would be changed.

That objection could probably be covered by adding speech patterns rather than eliminating them.

How about religion? Would the atheists here be comfortable with a coin toss approach to being religious or not?

I'm ethnically Jewish (personally agnostic). I'm uncomfortable with Christianity in a way which I think is different from the way people who were raised Christian and who've had bad experiences are. I haven't had personal bad experiences with Christianity, but I'm not only edgy about it, but it's like Gandhi and the murder pill-- I'm not comfortable with the idea of fading out my discomfort, even though I can't see that it's doing me any good.

Being Jewish carries a lot of memories with it. I only go to a service if it's an important event for someone else, but if I do, I'm reasonably familiar with the ritual. I still like the Jewish folk songs I learned in Hebrew school. I suppose I could keep all that (from this paragraph) if the coin toss came up Unitarian, but not otherwise. It's not as though the folk songs are a secret, but no one else especially bothers to learn them.

The general point is that these differences aren't just pasted-on labels for the most part. (The exception I'm thinking of is a news story I read about an anti-Semite in Eastern Europe who found out he had Jewish ancestry, gave up anti-Semitism, and became observant. People are very strange.)

I'm not sure how useful arguments from completely imaginary tech are.

Other free association: I've read about an exercise where people were asked to list the labels they identified with, and the list tended to mostly include things they'd been hurt about.

Comment author: Yvain 31 October 2011 09:51:10PM 1 point [-]

Religion seems different insofar as some people think it has a truth value. If I believe Jesus is the Son of God, that's a very strong argument for not using a machine that turns me into an atheist - once I'm an atheist, I would be wrong on the Jesus question.

I also am ethnically Jewish, but I don't consider that to be an interesting test case of the principle. Part of why I find the black case interesting is that black people could continue to perpetuate black culture even if they had white skin. Since Jews don't look very different from the majority population, it's unclear what a machine to make me "not Jewish" would mean other than that I lose Jewish culture and ritual and so on, which makes it a totally different case.

Write about risk and decision-making, win money

9 Yvain 31 October 2011 09:44PM

A company near me is running an "outreach competition" with a first prize of 200 euro (=$275) for essays on "risk" from high school and university students. I know the person running the competition and he's very interested in probability calibration and cognitive biases. You can read more about the competition here.

Comment author: SilasBarta 29 October 2011 04:09:48PM *  11 points [-]

Once again, I bring up the exchange from Gideon's Crossing, where a black doctor advises a deaf woman to give her deaf daughter cochlear implants:

Mother: You're saying that hearing people are better than deaf people!
Doctor: I'm saying it's easier.
Mother: Would your life be easier if you were white?

For those that don't catch the analogy: yes, life might be easier if you were hearing rather than deaf, but gaining the ability to hear would change a fundamental part of your identity and separate you from your "native culture".

Not endorsing this view, just trying to give a better intuition for it.

Comment author: Yvain 31 October 2011 06:22:00PM *  13 points [-]

It's interesting that obvious moral of your anecdote is supposed to be that no black person would want to change into a white person, even though life would be easier. I mean, I agree it's probably true, but it seems mysterious to me, like something that needs to be explained. One explanation is that a single black person changing to white would in a sense be betraying all her black friends, or legitimizing the idea that being black is worse than being white, but I can think of a contrived scenario where those explanations don't seem to apply.

Suppose that we had a machine that could change people's skin color, physical features, and speech patterns, so that people of one race could be turned indistinguishable from people of another. And suppose we wanted to end all racial discrimination forever by making everyone the same race. So we flip a coin to decide whether all white people have to change into black people, or all black people have to change into white people - discrimination disappears either way, and this way we know it's not a power thing where white people are trying to enforce their own norms.

I'm white, and I don't think I would object too much if the coin came up as "all whites have to change to blacks." I could see some white people objecting on aesthetic grounds, that they've been conditioned to think white people are more attractive and don't want to be in bodies they would view as less attractive. I could imagine a whole bunch of white people objecting just to be contrary. But overall I can't think of any really good objections from the white point of view.

But I know that part of white privilege is the privilege of thinking race doesn't make that much difference, so I predict that black people would want to think much harder about the case where all blacks have to change into whites. They'd probably have the same aesthetics and general contrariness objections as the white people, but if the rest of the thread is any indication there might also be an objection surrounding "black culture".

One could say that whatever black people like about black culture, they could continue to like if they had white skin. I guess the counterargument would be that black culture needs a certain critical mass to survive, and that if there were no artificial division between blacks and whites forcing them into different communities and different "meme pools", it would get overwhelmed by the more common white culture. But this seems like it's also a good argument against any attempt to fight racism or end segregation. And although I am almost sure someone is going to scoff really hard at me for saying this and explain why it's totally not appropriate, lots of white people seem to like a lot of black culture and even be pretty good at some "traditionally black" forms of expression, and vice versa.

I'd be really interested in hearing from some minority - whether in terms of race or sexual orientation or whatever - who wouldn't want her community to accept a coin toss on the principles described above.

Comment author: Yvain 25 October 2011 11:03:04PM *  45 points [-]

What are your favorite pieces of writing advice?

There's that quote about how "the most important thing is sincerity, and if you can fake that, you've got it made." So there are two equal and opposite commandments for popular writing. First, you've got to sound like you're chatting with your reader, like you're giving them an unfiltered stream-of-consciousness access to your ideas as you think them. Second, on no account should you actually do that.

Eliezer is one of the masters at this; his essays are littered with phrases like "y'know" and "pretty much", but they're way too tight to be hastily published first drafts (or maybe I'm wrong and Eliezer is one of the few people in the world who can do this; chances are you're not). You've got to put a lot of work into making something look that spontaneous. I'm a fan of words like "sorta" and "kinda" myself, but I have literally gone through paragraphs and replaced all of the "to some degrees" with "sortas" to get the tone how I wanted it.

I like inserting myself and my thought processes into things I write. It's a no-no in serious writing, but in informal writing it can emphasize the informality and become endearing, a sort of "we can take off the masks now, because we're all friends here". This only works if your personal asides are actually endearing to people, or at least not actively boring and off-putting, but if you get it right it lets you keep more spontaneity, since talking in first person is a natural impulse. As in everything, "first learn the rules and the reasons for them, then break them as much as you want".

The real meat of writing comes from an intuitive flow of words and ideas that surprises even yourself. Editing can only enhance and purify writing so far; it needs to have some natural potential to begin with. My own process here is to mentally rehearse an idea very many times without even thinking about writing. Once I'm an expert at explaining it to myself or an imaginary partner, then I transcribe the explanation I settle upon (some people say they don't think in words; I predict writing will not come naturally to these people). Then I edit the heck out of it.

The best way to improve the natural flow of ideas, and your writing in general, is to read really good writers so much that you unconsciously pick up their turns of phrase and don't even realize when you're using them. The best time to do that is when you're eight years old; the second best time is now.

Your role models here should be those vampires who hunt down the talented, suck out their souls, and absorb their powers. Which writers' souls you feast upon depends on your own natural style and your goals. I've gained most from reading Eliezer, Mencius Moldbug, Aleister Crowley, and G.K. Chesterton (links go to writing samples from each I consider particularly good); I'm currently making my way through Chesterton's collected works pretty much with the sole aim of imprinting his writing style into my brain.

Stepping from the sublime to the ridiculous, I took a lot from reading Dave Barry when I was a child. He has a very observational sense of humor, the sort where instead of going out looking for jokes, he just writes about a topic and it ends up funny. It's not hard to copy if you're familiar enough with it. And if you can be funny, people will read you whether you have any other redeeming qualities or not.

Getting imprinted with good writers like this will serve you for your entire life. It will serve you whether you're on your fiftieth draft of a thesis paper, or you're rushing a Less Wrong comment in the three minutes before you have to go to work. It will even serve you in regular old non-written conversation, because wit and clarity are independent of medium.

And it will also inform and limit your use of all the other rules above. Luke's fourth point - telling stories about characters taking actions - is a good one, but he very reasonably didn't start this post off with a story about some student working on a term paper. There have been a few LW posts that kind of seemed kludgy and artificial in adding characters and stories, and others that did it really well. Probably some very smart person could figure out why it succeeds somewhere and fails somewhere else, but it's easier to just cultivate the virtue that is nameless.

Some people say to write down everything and only edit later. I take the opposite tack. I used to believe that I rarely edited at all because I usually publish something as soon as it's done. Then a friend watching me write said that she was getting seasick from my tendency to go back and forth deleting and rewriting the same sentence fragment or paragraph before moving on. Most likely the best writers combine both editing methods.

Comment author: Vaniver 23 October 2011 06:51:44PM *  2 points [-]

I don't think the issue is that I can't tell I'm not learning anything - I think the issue is that I'm inclined to reward things other than teaching me things even in an environment ostensibly about learning.

One of the issues was that, even after Fox had been revealed as a fake, is that people were earnestly interested in the application of game theory to physician education. That is, the amount academics care about various ideas depends heavily on the panache with which those ideas are delivered.

Comment author: Yvain 23 October 2011 10:36:12PM 5 points [-]

I think it's about physician education, ie of doctors.

Comment author: Yvain 23 October 2011 08:32:27PM *  29 points [-]

Thank you for linking to the original paper. I read it, and I no longer think this was a completely valid experiment. Or, at least, it is not as strong as it seems to be.

The questionnaire that they gave subjects read:

  • Did he dwell upon the obvious?
  • Did he seem interested in his subject?
  • Did he use enough examples to clarify his material?
  • Did he present his material in a well-organized form?
  • Did he stimulate your thinking?
  • Did he put his material across in an interesting way?
  • Have you read any of this speaker's publications?

Few of these seem completely incompatible with illogical nonsense. I'm sure he sounded interested in his subject, and for all I know he used lots of examples and put things across in an interesting way (Something like "educating physicians is much like hunting tigers, because of the part with the stethoscopes" is interesting and provides examples, but still total nonsense).

Another section asked for written comments, and they received comments like "Enjoyed listening", "Has warm manner", "Good flow, seems enthusiastic", all of which I'm sure were true, as well as a few like "Too intellectual a presentation", "left out relevant examples", and my favorite, "He misses the last few phrases which I believe would have tied together his ideas for me." These last ones seem to me like face-saving ways of saying "I didn't actually understand the slightest bit of what he was talking about".

If you have a really nice, warm presenter, probably after a lot of old stuffy guys who have bored everyone in the conference to death, and you ask for evaluations that don't really ask the questions you're interested in but give enough waffle room to allow respondents to praise the presentation they don't understand, I'm not at all surprised that people would do that.

Why, oh why, couldn't the experimenters have included a simple "Did you or did you not understand what this man was talking about?" It almost seems suspicious, like they were worried they wouldn't get as interesting a result.

...or maybe it's just normal incompetence. I have this same problem with course evaluations in my own university: they consist entirely of closed questions on peripheral issues that force me to end up giving very positive evaluations to awful classes. For example, it might ask me to rate from 1 to 5 the answers to lots of questions like "Was the professor always available to help students?" and "Was the work load reasonable?" and other things I am forced to admit were satisfactory, but nothing like "Did the professor drone on in a monotone about his own research interests for two hours a day and never actually get to covering the course material?"

Comment author: Yvain 22 October 2011 05:18:58PM *  26 points [-]

Don't you feel in your heart that these contradictions do not really contradict: that there is a cosmos that contains them all? The soul goes round upon a wheel of stars and all things return; perhaps Strake and I have striven in many shapes, beast against beast and bird against bird, and perhaps we shall strive for ever. But since we seek and need each other, even that eternal hatred is an eternal love. Good and evil go round in a wheel that is one thing and not many. Do you not realize in your heart, do you not believe behind all your beliefs, that there is but one reality and we are its shadows; and that all things are but aspects of one thing: a centre where men melt into Man and Man into God?'

'No,' said Father Brown.

-- G.K. Chesterton

Comment author: PhilGoetz 22 October 2011 02:25:12AM *  2 points [-]

Suppose that you have one extra ticket to the Grand Galloping Gala, and you have several friends who each want it desperately. You can give it to only one of them. Doesn't the agonizing over that decision feel a lot like the agonizing over whether to buy or torrent? Yet we don't think of that as involving willpower.

Comment author: Yvain 22 October 2011 11:22:28AM *  3 points [-]

At the risk of totally reducing this to unsupportable subjective intuitions...no, the two decisions wouldn't feel the same at all.

I can think of some cases in which it would feel similar. If one of the ticket-seekers was my best friend whom I'd known forever, and another was a girl I was trying to impress, and I had to decide between loyalty to my best friend or personal gain from impressing the girl. Or if one of the ticket-seekers had an incurable disease and this was her last chance to enjoy herself, and the other was a much better friend and much more fun to be around. But both of these are, in some way, moral issues.

In the simple ticket-seeker case without any of these complications, there would be a tough decision, but it would be symmetrical: there would be certain reasons for choosing Friend A, and certain others for choosing Friend B, and I could just decide between them. In the torrenting case, and the complicated ticket-seeker cases, it feels asymmetrical: like I have a better nature tending toward one side, and temptation tending toward the other side. This asymmetry seems to be the uniting factor behind my feeling of needing "willpower" for some decisions.

A signaling theory of class x politics interaction

53 Yvain 17 October 2011 06:49PM

The media, most recently The Economist and Scientific American, have been publicizing a surprising statistical finding: in the current economic climate, when more Americans than ever are poor, support for policies that redistribute wealth to the poor are at their lowest levels ever. This new-found antipathy towards aid to the poor concentrates in people who are near but not yet on the lowest rung of the social ladder. The Economist adds some related statistics: those who earn slightly more than the minimum wage are most against raising the minimum wage, and support for welfare in an area decreases as the percentage of welfare recipients in the area rises.

Both articles explain the paradoxical findings by appealing to something called "last place aversion", an observed tendency for people to overvalue not being in last place. For example, in laboratory experiments where everyone gets randomly determined amounts of money, most people are willing to help those with less money than themselves gain cash - except the person with the second to lowest amount of money, who tends to try to thwart the person in last place even if it means enriching those who already have the most.

"Last place aversion" is interesting, and certainly deserves at least a footnote in the catalogue of cognitive biases and heuristics, but I find it an unsatisfying explanation for the observations about US attitudes toward wealth redistribution. For one thing, the entire point of last place aversion is that it only affects those in last place, but in a massive country like the United States, everyone can find someone worse off than themselves (with one exception). For another, redistributive policies usually stop short of making those who need government handouts wealthier than those who do not; subsidizing more homeless shelters doesn't risk giving the homeless a nicer house than your own. Finally, many of the policies people oppose, like taxing the rich, don't directly translate to helping those in last place.

I propose a different mechanism, one based on ... wait for it ... signaling.

In a previous post, I discussed multi-level signaling and counter-signaling, where each level tries to differentiate itself from the level beneath it. For example, the nouveau riche differentiate themselves from the middle class by buying ostentatious bling, and the nobility (who are at no risk of being mistaken for the middle class) differentiate themselves from the nouveau riche by not buying ostentatious bling.

The very poor have one strong incentive to support redistribution of wealth: they need the money. They also have a second, subtler incentive: most redistributive policies come packaged with a philosophy that the poor are not personally responsible for the poverty, but are at least partially the victims of the rest of society. Therefore, these policies inflate both their pocketbook and their ego.

The lower middle class gain what status they have by not being the very poor; effective status signaling for a lower middle class person is that which proves that she is certainly not poor. One effective method is to hold opinions contrary to those of the poor: that redistribution of wealth is evil and that the poor deserve their poverty. This ideology celebrates the superiority of the lower middle class over the poor by emphasizing the biggest difference between the lower middle class and the very poor: self-reliance. By asserting this ideology, a lower middle class person can prove her lower middle class status.

The upper middle class gain what status they have by not being the lower middle class; effective status signaling for an upper middle class person is that which proves that she is certainly not lower middle class. One effective way is to hold opinions contrary to those of the lower middle class: that really the poor and lower middle class are the same sort of people, but some of them got lucky and some of them got unlucky. The only people who can comfortably say "Deep down there's really no difference between myself and a poor person" are people confident that no one will actually mistake them for a poor person after they say this.

As a thought experiment, imagine your reactions to the following figures:

1. A bearded grizzled man in ripped jeans, smelling slightly of alcohol, ranting about how the government needs to give more free benefits to the poor.

2. A bearded grizzled man in ripped jeans, smelling slightly of alcohol, ranting about how the poor are lazy and he worked hard to get where he is today.

3. A well-dressed, stylish man in a business suit, ranting about how the government needs to give more free benefits to the poor.

4. A well-dressed, stylish man in a business suit, ranting about how the poor are lazy and he worked hard to get where he is today.

My gut reactions are (1, lazy guy who wants free money) (2, honorable working class salt-of-the-earth) (3, compassionate guy with good intentions) (4, insensitive guy who doesn't realize his privilege). If these are relatively common reactions, these would suffice to explain the signaling patterns in these demographics.

If this were true, it would explain the unusual trends cited in the first paragraph. An area where welfare became more common would see support for welfare drop, as it became more and more necessary for people to signal that they themselves were not welfare recipients. Support for minimum wage would be lowest among people who earn just slightly more than minimum wage, and who need to signal that they are not minimum wage earners. And since upper middle class people tend to favor redistribution as a status signal and lower middle class people tend to oppose it, a recession that drives more people into the lower middle class would cause a drop in support for redistributive policies.

Calibrate your self-assessments

68 Post author: Yvain 09 October 2011 11:26PM

When I moved to Ireland, I knew that their school system, and in particular their examinations, would be different from the ones I was used to. I educated myself on them and by the time I took my first exam I thought I was reasonably prepared.

I walked out of my first examination almost certain I had failed. I remember emailing my parents, apologizing to them for my failure and promising I would do better when I repeated the class.

Then I got my results back, and learned I had passed with honors.

This situation repeated itself with depressing regularity over the next few semesters. Took exam, walked out in tears certain I had failed, made angsty complaints and apologies, got results back, celebrated. Eventually I decided that I might as well skip steps two to five and go straight to the celebrations.

This was harder than I expected. Just knowing that my feelings of abject failure usually ended out all right did not change those feelings of abject failure. I still walked out of each exam with the same gut certainty of disaster I had always had. What I did learn to do was ignore it: to force myself to walk home with a smile on my face and refuse to let myself dwell on the feelings of failure or take them seriously. And in this I was successful, and now the feelings of abject failure produce only a tiny twinge of stress.

In LW terminology, I am calibrating my self-assessment of examination success1.

We appreciate objective measurements, like a percent score on an examination, or the running time of a marathon in minutes and seconds. But in the absence of such measurements, we use subjective mental estimates: how I feel I did on this exam, or how plausible that theory sounds.

The rationality literature has especially focused on one particular subjective mental estimate: our feelings of probability. For example, someone may say they feel 80% certain that Germany is larger than France. However, if they consistently answer questions like this with 80% confidence, and only get 60% right, then we say they are mis-calibrated: their subjective mental estimate of probability has a consistent mismatch with a more normatively correct probability. Calibration means revising your subjective mental estimate until it matches the objective value it tries to estimate; so that when you estimate something with 80% confidence, you get it right 80% of the time.

My story about exam scores is also a story about calibration. My subjective mental estimate of my exam scores was consistently too low; I would estimate I failed when I had really passed by a wide margin. By suppressing my original mental estimate and replacing it with one better informed by past experience, I am calibrating my estimate of exam scores.

Since passing my exams, I've identified other areas of my life where I need to calibrate my estimates:

-- Embarrassment. I used to be mortified if I answered a question wrong in class, assuming that people would judge me on it as long as they knew me. After thinking about it, I realized that although many people in my class answer questions wrong every day, I literally cannot remember a single one. If you pointed out any student in my class, even one of my close friends who I would be expected to pay extra attention to, and asked me "Has this person ever answered a question wrong in class?" I wouldn't be able to tell you. This suggests they won't remember my mistakes either, and that my subjective feeling of loss of respect on answering a question wrong is exaggerated to say the least2.

-- Interestingness. I tend to think that if I talk about something I'm interested in, other people will be interested in it too. No matter how fascinating the underlying concept to me, nor how well I think I'm explaining it, this almost never happens.

-- Flirting. Through painful trial and error, I've found that my hunch that a woman likes me is almost always wrong. Someone will be flirting very heavily with me, and I'll think "there is no way in the world she's not into me", and then it will turn out she will not be into me.

These aren't just things I'm often wrong about; making a list of those would be a Sisyphean task. They're the things that I'm wrong about that my natural instincts never auto-correct, so that I know I'm going to keep being wrong unless I consciously calibrate my natural instincts against a reasoned opinion.

In general, I find I am most often miscalibrated in areas that relate to self evaluation. Cognitive psychology has a slew of ideas about so-called "self-assessment biases". You've probably heard the self-serving ones where 94% of professors rate their teaching ability above average, or how everyone thinks they're an above average driver, or how (ironically) everyone thinks they're less susceptible to biases than other people. But more surprisingly, I also find cases where people consistently underestimate themselves - like my own tendency to always think I've failed my examinations. I don't have a good explanation of this - I don't know if it's strategic humility, self-verification, some underlying depression-like state, or what - but I'm pretty sure it exists. And there are two situations in which I find it most common and most annoying.

The first involves good looks. Some people just have no idea how attractive they are or aren't. This is most obvious in body dysmorphic disorder, a condition where normal looking (or even very attractive) people somehow get it into their head that some feature of theirs - their nose, their hair, their weight - is inhumanly hideous and that they look like some kind of swamp monster. This is an officially recognized psychiatric disorder because it's completely divorced from reality - usually their nose or hair or whatever looks absolutely normal and just like everyone else's.

BDD is less rare than people think, at about one to two percent of the population, but even people without the full-fledged disorder can be really bad at determining how attractive they are or aren't. There are a lot of pretty girls who go around saying they're ugly in order to trick people into complimenting them, or to signal that they're available and not too picky, but I've come to realize that there are also a lot of pretty girls who genuinely believe they're ugly (it's less obvious in men, but I wouldn't be surprised if it were there under the surface).

And research agrees: studies show that people are uniquely bad at rating their own physical attractiveness. The opinion of unbiased observers evaluating a subject's attractiveness usually correlate at a level of r = .4 to .5; the opinion of the subject herself correlates with everyone else only around the r = .2 level. Other studies using purportedly "objective" measures of attractiveness like facial symmetry report a similarly low level of correlation between the objective measures and the self-reports. What self-reported physical attractiveness correlates strongly with is not objective attractiveness, but self-reported self-esteem, with r values around .5 or .6 depending on the study.

If you're not so good at statistics - that means that people often agree on how attractive a particular subject is, but that subject's estimate of her own attractiveness is often completely different from everyone else's (in either direction), and more related to that subject's self-esteem than to reality.

Sites like hotornot.com or okcupid's MyBestFace have a lot of problems, most obviously that they depend a lot upon how good a specific photo is. But I think either is leagues ahead of trying to guess how attractive you are to others based on how attractive you feel. If you have any concern whatsoever about how attractive you are, the worst thing you can do is trust your own brain, especially if it's telling you you're probably pretty ugly when everyone around you seems to think you're okay.

Which brings me to the number one most tragic failure of the inside view I see in my friends, my acquaintances, and the psychiatric patients I encounter.

Nietzsche said that a casual stroll through an insane asylum shows that faith does not prove anything. Such an experience might also teach people to be skeptical of their own subjective valuation of themselves - their self-esteem. If our hypothetical visitor doesn't figure it out after seeing the depressed patients, who are obsessed with their own guilt and moral worthlessness to the point of confessing to any crime they hear about because it seems like the sort of thing someone as awful as themselves might do, she can go visit the schizophrenics with delusions of grandeur, who insist they are the next Jesus or Einstein, or God's chosen representative on Earth.

These people get locked away because their self-esteem is at an extreme no sane human would ever reach. But your location outside the insane asylum doesn't prove your own calculations of self-esteem come from reasoning processes that are any more valid. We all know self-obsessed narcissists without any real achievements to their name, and we all know people who insist that they are ugly and stupid and unlikeable even though they don't seem any worse off than anyone else.

Research confirms that people's self-esteem is poorly correlated with reality. Across many experiments with many different designs, people's self-reported likeability has no correlation with their likeability as reported by other people with whom they interact. This is true whether the experiment measures artificial interaction in a lab, simulated "dates" with people of the opposite sex, or the attitudes of their roommates.

There are no studies correlating self-reported morality with experimentally determined morality, but if you want conduct one, you could probably gather enough secretly gay evangelical ministers and adulterous family-values politicians to make up a pretty good sample size.

If you hate yourself and think you're worthless, take a moment to consider whether you have any evidence that you're objectively doing any worse than anyone else, or whether you just have a low self-esteem set point. If the latter appears to be true, then try to replace the inside view with the outside view when worrying about how much bother you're being to other people or whether you "deserve" to be happy.

(if your problem is in the other direction you may not have as much vested interest in correcting yourself, but do keep in mind that most of the purported benefits of self-confidence have been exaggerated).

SUMMARY


People's subjective mental estimates are often way off, especially when they're estimating qualities closely linked to their self-worth. Both everyday experience and scientific research provide ample evidence of people who both underestimate and overestimate themselves in various ways. If you worry you may be one of those people, try and get objective estimates of the parameter you're concerned about from other people or from empirical testing. Then make an effort of will to consciously replace your subjective estimates with your new better-calibrated estimates.

 

FOOTNOTES

1: This could also be interpreted as replacing the Inside View with the Outside View and this would also be a good moral to draw from the story; I'm phrasing it in terms of calibration because it's more appropriate for some of the other examples later down.

2: See Gilovich, Medvec, and Savitsky, 2000 for experimental proof of the same idea

Comment author: wedrifid 16 October 2011 06:42:06PM 2 points [-]

I'm not sure what I think of this argument, because small differences in intelligence can have major differences in "personhood" - for example, I can't even name a number of cows it would take such that I would be equally comfortable killing X cows as one person.

I believe you - for as long as you are typing at a keyboard. Once we put it to the test and you are forced out into a field with a knife and blood pouring down your arm your 'comfort' would become far more measurable.

Comment author: Yvain 16 October 2011 09:04:28PM 11 points [-]

Really? I would have gone the opposite way - utilitarianism seems to tell me that if cows have nonzero value then a certain number of them must add up to one human, but I have much stronger mental conditioning against killing humans than cows.

Comment author: Yvain 16 October 2011 12:25:40PM *  8 points [-]

I can't remember where I saw it (I hope it wasn't on here), but someone recommended that someone with vegetarian sympathies could do some good just by switching from chicken to beef. The idea is that if you're eating a constant amount of meat by weight, you have to kill a couple hundred chickens to get the same amount of meat as killing one cow. If you don't believe there's a significant "personhood" difference between cows and chickens, that's cutting your death toll by orders of magnitude.

I'm not sure what I think of this argument, because small differences in intelligence can have major differences in "personhood" - for example, I can't even name a number of cows it would take such that I would be equally comfortable killing X cows as one person. That means that even a small intelligence difference between cows and chickens could more than cancel out the gains from extra weight.

Really, what we need is some sort of animal which is both very large and very stupid. If only a stray prehistoric asteroid hadn't killed off our ideal food source.

Comment author: Yvain 15 October 2011 11:09:32AM 16 points [-]

"Critical thinking" is a useful phrase sometimes. It's a real term, and people usually have good associations with it. Saying "we need to teach people better critical thinking skills" usually elicits nods; "we need to teach people to be more rational" is more likely to get them grabbing their guns and heading to the hills.

Comment author: Morendil 14 October 2011 10:42:31PM *  12 points [-]

"When you see teenagers and young adults posting their interests in music, books, and film on their MySpace websites, consider the costly signaling principles at work."

Wait, what? What's costly about posting something to Myspace? The "costly" part of "costly signaling" has a rather precise meaning in evolutionary biology: an extravagant expenditure of scarce resources, useful because potential mates who correctly interpret that as entailing your high fitness thereby improve their own. Stotting is a typical example.

Here it seems to me that the author is just throwing scientific-sounding lingo around to bolster pure speculation. Taboo "costly signaling" and the author is saying: "the genes of girls who picked boys who spouted weird antisocial nonsense fared better, because only boys whose splendid health allowed them to pick up such nonsense in the first place would do that".

I'm unconvinced: stripped of the highfalutin' language this makes much less sense. Even if I try to repair the argument by bringing the parasite connection back in: "oh, this boy is quoting David Lynch, cue some social module which infers that he must have spent time with people from foreign tribes, which means that he will have deliberately exposed himself to lice of unusual size, which means that he must be quite confident of his own health to start with; surely he will make a fine father for my offspring".

More abstractly, "costly signaling" theory requires that it should be hard for low-fitness individuals to send a fake signal. Clearly this isn't the case here - anyone can pretend to spout weird antisocial nonsense.

Comment author: Yvain 15 October 2011 11:02:57AM *  18 points [-]

More abstractly, "costly signaling" theory requires that it should be hard for low-fitness individuals to send a fake signal. Clearly this isn't the case here - anyone can pretend to spout weird antisocial nonsense.

I disagree; I think this is an excellent use of the handicap principle (though it remains to be seen whether it's actually true). Any gazelle can go stotting, too, if by "can" you mean "it is physically possible for them to do so". But only a gazelle very confident in its speed actually will, because all the other gazelles are too worried about being eaten by lions to dare to try it.

Likewise, anyone can claim to like weird foreign films on MySpace, but only a person very confident in her popularity actually will, because all the other people are too worried about being shunned.

The number one objection I have to this idea isn't that it's not evolutionarily plausible, but that it doesn't fit observed data: it's not the high school quarterbacks and cheerleaders who are liking weird foreign films, it's kinda weird people who aren't popular anyway: anime is the most obvious example. Any further argument for this idea would have to explain why it goes so terribly wrong.

Comment author: epigeios 12 October 2011 09:01:00PM *  1 point [-]

My interpretation of the post in this case is: it's not that you're not employing willpower, instead you're not employing personal morality. So, while TORRENT vs BUY fits into the societal ethics view, it does not fit into your personal morality.

From the personal morality perspective, the bad feeling you get is the thing you need willpower to fight against/suppress. You probably also need willpower to fight against/suppress the bad feeling you might be getting from buying the album. These need not be mutually exclusive. Personal morality can be both against torrenting and against spending money unduly.

Comment author: Yvain 12 October 2011 09:39:17PM 4 points [-]

Let me rephrase my objection, then.

I feel a certain sense of mental struggle when considering whether to torrent music. I don't feel this same sense of mental struggle when considering whether or not to murder or steal or cheat . Although both of these are situations that call my personal morality, the torrenting situation seems to be an interesting special case.

We need a word to define the way in which the torrenting situation is a special case and not just another case where I don't murder or steal or cheat because I'm not that kind of person. The majority of the English-speaking world seems to use "willpower". As far as I know there's no other definition of willpower, where we could say "Oh, that's real willpower, this torrenting thing is something else." If we didn't have the word "willpower", we'd have to make up a different word, like "conscious-alignment in mental struggle" or something.

So why not use the word "willpower" here?

Comment author: Yvain 11 October 2011 06:31:54PM *  0 points [-]

25% or so probability I'll be there. More if someone else plans to drive through Clonmel and wants to transport me.

Comment author: Brickman 10 October 2011 01:19:17AM 6 points [-]

I like the first two, and the chess one's pretty interesting though I can't imagine I'd have an easy time getting someone to stand still long enough to hear the whole thing as an argument. But I don't really like the last one. You've been tricked into accepting his premise, that death lets you create more meaningful art, and trying to regain ground from there. It's that premise itself that you should be arguing against--point out all the great literature and art that isn't about death, and that you could still have all of that once death was gone. Also point out that to someone with cancer today the availability of art is probably less valuable than the availability of a cure would be, and there's no reason to assume that'll change if you double his age, even if you double it several times.

Comment author: Yvain 10 October 2011 05:43:51PM 15 points [-]

Also point out that to someone with cancer today the availability of art is probably less valuable than the availability of a cure would be.

Or to approach the same point from a slightly different direction - Elie Wiesel wrote some pretty awesome stuff, but that doesn't mean we should have more Holocausts.

Calibrate your self-assessments

68 Post author: Yvain 09 October 2011 11:26PM

When I moved to Ireland, I knew that their school system, and in particular their examinations, would be different from the ones I was used to. I educated myself on them and by the time I took my first exam I thought I was reasonably prepared.

I walked out of my first examination almost certain I had failed. I remember emailing my parents, apologizing to them for my failure and promising I would do better when I repeated the class.

Then I got my results back, and learned I had passed with honors.

This situation repeated itself with depressing regularity over the next few semesters. Took exam, walked out in tears certain I had failed, made angsty complaints and apologies, got results back, celebrated. Eventually I decided that I might as well skip steps two to five and go straight to the celebrations.

This was harder than I expected. Just knowing that my feelings of abject failure usually ended out all right did not change those feelings of abject failure. I still walked out of each exam with the same gut certainty of disaster I had always had. What I did learn to do was ignore it: to force myself to walk home with a smile on my face and refuse to let myself dwell on the feelings of failure or take them seriously. And in this I was successful, and now the feelings of abject failure produce only a tiny twinge of stress.

In LW terminology, I am calibrating my self-assessment of examination success1.

We appreciate objective measurements, like a percent score on an examination, or the running time of a marathon in minutes and seconds. But in the absence of such measurements, we use subjective mental estimates: how I feel I did on this exam, or how plausible that theory sounds.

The rationality literature has especially focused on one particular subjective mental estimate: our feelings of probability. For example, someone may say they feel 80% certain that Germany is larger than France. However, if they consistently answer questions like this with 80% confidence, and only get 60% right, then we say they are mis-calibrated: their subjective mental estimate of probability has a consistent mismatch with a more normatively correct probability. Calibration means revising your subjective mental estimate until it matches the objective value it tries to estimate; so that when you estimate something with 80% confidence, you get it right 80% of the time.

My story about exam scores is also a story about calibration. My subjective mental estimate of my exam scores was consistently too low; I would estimate I failed when I had really passed by a wide margin. By suppressing my original mental estimate and replacing it with one better informed by past experience, I am calibrating my estimate of exam scores.

Since passing my exams, I've identified other areas of my life where I need to calibrate my estimates:

-- Embarrassment. I used to be mortified if I answered a question wrong in class, assuming that people would judge me on it as long as they knew me. After thinking about it, I realized that although many people in my class answer questions wrong every day, I literally cannot remember a single one. If you pointed out any student in my class, even one of my close friends who I would be expected to pay extra attention to, and asked me "Has this person ever answered a question wrong in class?" I wouldn't be able to tell you. This suggests they won't remember my mistakes either, and that my subjective feeling of loss of respect on answering a question wrong is exaggerated to say the least2.

-- Interestingness. I tend to think that if I talk about something I'm interested in, other people will be interested in it too. No matter how fascinating the underlying concept to me, nor how well I think I'm explaining it, this almost never happens.

-- Flirting. Through painful trial and error, I've found that my hunch that a woman likes me is almost always wrong. Someone will be flirting very heavily with me, and I'll think "there is no way in the world she's not into me", and then it will turn out she will not be into me.

These aren't just things I'm often wrong about; making a list of those would be a Sisyphean task. They're the things that I'm wrong about that my natural instincts never auto-correct, so that I know I'm going to keep being wrong unless I consciously calibrate my natural instincts against a reasoned opinion.

In general, I find I am most often miscalibrated in areas that relate to self evaluation. Cognitive psychology has a slew of ideas about so-called "self-assessment biases". You've probably heard the self-serving ones where 94% of professors rate their teaching ability above average, or how everyone thinks they're an above average driver, or how (ironically) everyone thinks they're less susceptible to biases than other people. But more surprisingly, I also find cases where people consistently underestimate themselves - like my own tendency to always think I've failed my examinations. I don't have a good explanation of this - I don't know if it's strategic humility, self-verification, some underlying depression-like state, or what - but I'm pretty sure it exists. And there are two situations in which I find it most common and most annoying.

The first involves good looks. Some people just have no idea how attractive they are or aren't. This is most obvious in body dysmorphic disorder, a condition where normal looking (or even very attractive) people somehow get it into their head that some feature of theirs - their nose, their hair, their weight - is inhumanly hideous and that they look like some kind of swamp monster. This is an officially recognized psychiatric disorder because it's completely divorced from reality - usually their nose or hair or whatever looks absolutely normal and just like everyone else's.

BDD is less rare than people think, at about one to two percent of the population, but even people without the full-fledged disorder can be really bad at determining how attractive they are or aren't. There are a lot of pretty girls who go around saying they're ugly in order to trick people into complimenting them, or to signal that they're available and not too picky, but I've come to realize that there are also a lot of pretty girls who genuinely believe they're ugly (it's less obvious in men, but I wouldn't be surprised if it were there under the surface).

And research agrees: studies show that people are uniquely bad at rating their own physical attractiveness. The opinion of unbiased observers evaluating a subject's attractiveness usually correlate at a level of r = .4 to .5; the opinion of the subject herself correlates with everyone else only around the r = .2 level. Other studies using purportedly "objective" measures of attractiveness like facial symmetry report a similarly low level of correlation between the objective measures and the self-reports. What self-reported physical attractiveness correlates strongly with is not objective attractiveness, but self-reported self-esteem, with r values around .5 or .6 depending on the study.

If you're not so good at statistics - that means that people often agree on how attractive a particular subject is, but that subject's estimate of her own attractiveness is often completely different from everyone else's (in either direction), and more related to that subject's self-esteem than to reality.

Sites like hotornot.com or okcupid's MyBestFace have a lot of problems, most obviously that they depend a lot upon how good a specific photo is. But I think either is leagues ahead of trying to guess how attractive you are to others based on how attractive you feel. If you have any concern whatsoever about how attractive you are, the worst thing you can do is trust your own brain, especially if it's telling you you're probably pretty ugly when everyone around you seems to think you're okay.

Which brings me to the number one most tragic failure of the inside view I see in my friends, my acquaintances, and the psychiatric patients I encounter.

Nietzsche said that a casual stroll through an insane asylum shows that faith does not prove anything. Such an experience might also teach people to be skeptical of their own subjective valuation of themselves - their self-esteem. If our hypothetical visitor doesn't figure it out after seeing the depressed patients, who are obsessed with their own guilt and moral worthlessness to the point of confessing to any crime they hear about because it seems like the sort of thing someone as awful as themselves might do, she can go visit the schizophrenics with delusions of grandeur, who insist they are the next Jesus or Einstein, or God's chosen representative on Earth.

These people get locked away because their self-esteem is at an extreme no sane human would ever reach. But your location outside the insane asylum doesn't prove your own calculations of self-esteem come from reasoning processes that are any more valid. We all know self-obsessed narcissists without any real achievements to their name, and we all know people who insist that they are ugly and stupid and unlikeable even though they don't seem any worse off than anyone else.

Research confirms that people's self-esteem is poorly correlated with reality. Across many experiments with many different designs, people's self-reported likeability has no correlation with their likeability as reported by other people with whom they interact. This is true whether the experiment measures artificial interaction in a lab, simulated "dates" with people of the opposite sex, or the attitudes of their roommates.

There are no studies correlating self-reported morality with experimentally determined morality, but if you want conduct one, you could probably gather enough secretly gay evangelical ministers and adulterous family-values politicians to make up a pretty good sample size.

If you hate yourself and think you're worthless, take a moment to consider whether you have any evidence that you're objectively doing any worse than anyone else, or whether you just have a low self-esteem set point. If the latter appears to be true, then try to replace the inside view with the outside view when worrying about how much bother you're being to other people or whether you "deserve" to be happy.

(if your problem is in the other direction you may not have as much vested interest in correcting yourself, but do keep in mind that most of the purported benefits of self-confidence have been exaggerated).

SUMMARY


People's subjective mental estimates are often way off, especially when they're estimating qualities closely linked to their self-worth. Both everyday experience and scientific research provide ample evidence of people who both underestimate and overestimate themselves in various ways. If you worry you may be one of those people, try and get objective estimates of the parameter you're concerned about from other people or from empirical testing. Then make an effort of will to consciously replace your subjective estimates with your new better-calibrated estimates.

 

FOOTNOTES

1: This could also be interpreted as replacing the Inside View with the Outside View and this would also be a good moral to draw from the story; I'm phrasing it in terms of calibration because it's more appropriate for some of the other examples later down.

2: See Gilovich, Medvec, and Savitsky, 2000 for experimental proof of the same idea

Comment author: quinsie 09 October 2011 01:20:25AM *  0 points [-]

It all depends on why you decide to torrent/not torrent:

Are you more likely to torrent if the album is very expensive, or if it is very cheap? If you expect it to be of high quality, or of low quality? If the store you could buy the album at is far away, or very close? If you like the band that made it, or if you don't like them? Longer albums or shorter? Would you torrent less if the punishment for doing so was increased? Would you torrent more if it was harder to get caught? What if you were much richer, or much poorer?

I'm confident that if you were to analyze when you torrent vs. when you buy, you'd notice trends that, with a bit of effort, could be translated into a fairly reasonable "Will I Torrent or Buy?" function that predicts whether you'll torrent or not with much better accuracy than random.

Comment author: Yvain 09 October 2011 01:34:59AM 2 points [-]

I'm sure that's true, but what relevance does that have to the current discussion?

Comment author: Yvain 08 October 2011 06:01:23PM *  14 points [-]

So, when I agonize over whether to torrent an expensive album instead of paying for it, and about half the time I end up torrenting it and feeling bad, and about half the time I pay for it but don't enjoy doing so ... what exactly am I doing in the latter case if not employing willpower?

I mean, I know willpower probably isn't a real thing on the deepest levels of the brain, but it's fake in the same way centrifugal force is fake, not in the way Bigfoot is fake. It sure feels like I'm using willpower when I make moral decisions about pirating, and I don't understand how your model above interprets that.

Granted, there are many other moral decisions I make that don't require willpower and do conform to your model above, and if I had to choose black-and-white between ethics-as-willpower or ethics-as-choice I'd take the latter, your model just doesn't seem complete.

Comment author: Yvain 07 October 2011 03:08:56PM *  1 point [-]

See also http://www.physorg.com/news204872017.html (authored by a high school friend of mine, interestingly enough)

In response to comment by Yvain on On self-deception
Comment author: irrational 05 October 2011 09:11:04PM 2 points [-]

You have a very compelling point and I have to think about it. But there is meta-reasoning involved which is really tricky. As I start to read the book, I have some P(zoroastrianism is true). It's non-zero. Now I read the first chapter, it has some positive evidence for Z in it. I expected to see some evidence, but it is actual evidence which I have not previously considered. Should I adjust my P(Z is true) up? I think I must. So, if the book has many chapters, I must either get close to 1, or else start converging to some p < 1. Are you arguing for the latter?

Comment author: Yvain 06 October 2011 06:58:10PM *  3 points [-]

Consider the case where a friend says he saw a UFO. There are two possibilities: either the friend is lying/insane/gullible, or UFOs are real (there are probably some other possibilities, but for the sake of argument let's focus on these).

Your friend's statement can have different effects depending on what you already believe. If either probability is already at ~100%, you have no more work to do. IE, if you're already sure your friend is a liar, you dismiss this as yet another lie and don't start believing in UFOs; if you're already sure UFOs exist, you dismiss this as yet another UFO and don't start doubting your friend.

If you're not ~100% sure of either statement, then your observation will increase both the probability that your friend is a liar, and that aliens exist, but in different amounts. If you think your friend usually tells the truth, but you're not sure, it will increase your probability of UFOs quite a bit (your friend wouldn't lie to you!) but as long as you're not going to be sure of UFOs, you also have to leave some room for the case where UFOs aren't real, in which case the statement increases your probability that your friend is a liar.

When you hear a great argument for P, your pre-existing beliefs determine what you do in the same way as in the UFO example. It could mean that your interlocutor is a rhetorical genius so brilliant they can think up great arguments even for false positions. Or it could mean P is true. In real life, the probability of the interlocutor being such a rhetorical genius is always less than ~100%, meaning that it has to increase your probability of P at least a little.

In your example, we already know that the AI is a rhetorical genius who can create an arbitrarily good argument for anything. That totally explains away the brilliant arguments, leaving nothing left to be explained by Zoroastrianism actually being true. It's like when your friend who is a known insane liar says he saw a UFO: the insane liar part already explains away the evidence, so even though you're hearing words that sound like evidence, no probabilities are actually being shifted.

In response to comment by Yvain on On self-deception
Comment author: antigonus 05 October 2011 09:51:44PM *  12 points [-]

You may want to look at Brandon Fitelson's short paper Evidence of evidence is not (necessarily) evidence. You seem to be arguing that, since we have strong evidence that the book has strong evidence for Zoroastrianism before we read it, it follows that we already have (the most important part of) our evidence for Zoroastrianism. But it turns out that it's extremely tricky to make this sort of reasoning work. To use the most primitive example from the paper, discovering that a playing card C is black is evidence that C is the ace of spades. Furthermore, that C is the ace of spades is excellent evidence that it's an ace. But discovering that C is black does not give you any evidence whatsoever that C is an ace.

The problem here - at least one of them - is that discovering C is black is just as much evidence for C being the x of spades for any other card-value x. Similarly, before opening the book on Zoroastrianism, we have just as much evidence for the existence of strong evidence for Christianity/atheism/etc, so our credences shouldn't suddenly start favoring any one of these. But once we learn the evidence for Zoroastrianism, we've acquired new information, in just the same way that learning that the card is an ace of spades provides us new information if we previously just knew it was black.

I do suspect that there are relevant disanalogies here, but don't have a very detailed understanding of them.

Comment author: Yvain 06 October 2011 06:24:27PM *  2 points [-]

You seem to be arguing that, since we have strong evidence that the book has strong evidence for Zoroastrianism before we read it, it follows that we already have (the most important part of) our evidence for Zoroastrianism.

Not exactly. I do think this would be a true statement, if the book was a genuine book on Zoroastrianism and not a book which we know was designed to deceive us. But as far as I know it's only tangentially connected to the argument I'm making.

You may want to look at Brandon Fitelson's short paper Evidence of evidence is not (necessarily) evidence.

Thanks for summarizing the paper; I tried to read it but it was written in a way that seemed designed to be as obscure as possible. Your explanation makes more sense.

But I still don't see the problem. Learning a card is black increases the chance it's the ace of spades or clubs, but decreases the chance it's the ace of hearts or diamonds. The chance that it's the ace of spades becomes greater, but the net chance that it's an ace remains exactly the same. Evidence of evidence is still evidence, but evidence of evidence plus evidence of evidence that goes the opposite direction cancel out and make zero evidence.

Again, I'm not sure about the relevance here. It's not the case that, merely by knowing the book exists without reading it, we have new evidence for the existence of some evidence which both supports and, in a different way, opposes Zoroastrianism.

In response to On self-deception
Comment author: Yvain 05 October 2011 08:50:06PM *  24 points [-]

If before you open the book, you believe that the book will provide incredibly compelling evidence of Zoroastrianism whether or not Zoroastrianism is true, and upon opening the book you find incredibly compelling evidence of Zoroastrianism, your probability of Zoroastrianism should not change, since you didn't observe any evidence which is more likely to exist if Zoroastrianism were true than if it were not true.

It may be that you are underestimating the AI's cleverness, so that you expect to see decent evidence of Zoroastrianism, but in fact you found incredible evidence of Zoroastrianism, and so you become convinced. In this case your false belief about the AI not being too convincing is doing the philosophical work of deceiving you, and it's no longer really deceiving yourself. Deceiving yourself seems to be more about starting with all correct beliefs, but talking yourself into an incorrect belief.

If you happen to luck out into having a false belief about the AI being unconvincing, and if this situation with the library of theology just falls out of the sky without your arranging it, you got lucky - but that's being deceived by others. If you try to set up the situation, you can't deliberately underestimate the AI because you'll know you're doing it. And you can't set up the theological library situation until you're confident you've deliberately underestimated the AI.

Comment author: Morendil 05 October 2011 06:54:24PM 1 point [-]

Would appreciate some citations...

Comment author: Yvain 05 October 2011 08:26:41PM *  4 points [-]

The famous story here is that of Henry Beecher, usually considered discoverer of the placebo. From the Telegraph:

The power of the placebo first came to light during the Second World War. Morphine was in short supply in military field hospitals and an American anaesthetist called Henry Beecher, who was preparing to treat a soldier with terrible injuries, feared that without the drug the operation could induce a fatal heart attack. In desperation, one of the nurses injected the man with a harmless solution of saline. To Beecher’s surprise the patient settled down as if he had been given morphine and felt little pain during the operation.

Or for a more formal study, see Analgesic responses to morphine and placebo in individuals with postoperative pain

Comment author: Yvain 05 October 2011 06:04:36AM *  21 points [-]

One of the strongest cases for placebo effect is IV morphine, whose powerful painkilling effects can in many cases be simulated by IV saline which someone says is morphine. Finding the placebo effect with a professionally-administered IV drug usually given in hospital suggests it's not just an issue of compliance.

In response to Rationality Drugs
Comment author: Kevin 02 October 2011 12:19:05AM *  8 points [-]

Lately I've been extraordinarily surprised at how effective potassium and potassium salt are. By which I mean that simple potassium is probably the most positively mind altering supplement I've ever tried.

In response to comment by Kevin on Rationality Drugs
Comment author: Yvain 02 October 2011 08:17:04AM 2 points [-]

In what form did you take the potassium?

In response to comment by [deleted] on Rationality Drugs
Comment author: Douglas_Knight 01 October 2011 01:58:37PM *  2 points [-]

It's possible that some people with symptoms of depression are actually sleep deprived

I'm skeptical of this. Yes, five hours of sleep is bad for your mental health, but usually in a different direction. Did you have depressive symptoms that year? A key symptom of depression is lack of willpower - depressives don't normally have the willpower not to sleep. Quite the opposite, they sleep more the than normal. This would solve simple sleep deprivation. It's possible that they lack something more specific that normal people are able to get by sleeping, but even that does not sound terribly likely to me.

ETA: As various people comment, this is largely backwards. I particularly regret suggesting that people who spend a lot of time in bed get useful sleep. So maybe sleep deprivation contributes to some of the symptoms of depression. But there are other symptoms and I am skeptical that the two are confused.

Comment author: Yvain 01 October 2011 03:04:37PM *  6 points [-]

Depressed people can have either insomnia or hypersomnia; insomnia is significantly more common. Depression-related insomnia is usually "terminal" - people wake up very early and can't get back to sleep.

Strangely enough, there have been some studies showing that depriving depressed people of sleep has a strong positive effect on their mood, but of course then they're too sleep-deprived to enjoy it.

In response to Rationality Drugs
Comment author: Yvain 01 October 2011 02:51:43PM 16 points [-]

I've been self-experimenting with piracetam the past few months.

I usually study from a site called USMLEWorld with a selection of difficult case-based medical questions. For example, it might give a short story about a man coming into a hospital with a certain set of symptoms, and explain a little about his past medical history, and then ask multiple choice questions about what the most likely diagnosis is, or what medication would be most helpful. These are usually multi-step reasoning questions - for example, they might ask what side effect a certain patient could expect if given the ideal treatment for his disease, and before answering you need to determine what disease he has, what's the ideal treatment, and then what side effects that treatment could cause. My point is they're complicated (test multiple mental skills and not just simple recall) and realistic (similar to the problems a real doctor would encounter on the job).

I've tried comparing my performance on these questions on versus off piracetam. My usual procedure is to do twenty questions, take 2400 mg piracetam + 600 mg lecithin-derived choline, go do something fun and relaxing for an hour (about the time I've been told it takes for piracetam to take effect) then do twenty more questions. It's enough of a pain that I usually don't bother, but in about three months of occasionally doing my study this way I've got a pool of 160 questions on piracetam and 160 same-day control questions. Medicine is a sufficiently large and complicated field that I don't think three months worth of practice effects are a huge deal, and in any case I made sure to do equal piracetam and control questions every day so there wouldn't be a practiced-unpracticed confounder.

I got an average of 65% of questions right in the control condition and 60% of questions right on piracetam, but the difference was not significant.

USMLEWorld also tells you how other people did on each question; I used this information to run a different analysis controlling for the random difficulty variation in the questions. In the control condition I did 2.8% better than average, in the piracetam condition I did 1.3% worse than average; this wasn't a significant difference either.

I do worry that fatigue effects might have played a part; I tried to always rest and relax between conditions, but I was always doing piracetam after control (I wanted to have same-day comparisons to eliminate practice effects, and piracetam lasts too long for me to feel comfortable taking it first and then doing control after it wore off). But I didn't feel fatigued, and I haven't noticed huge fatigue effects when I study a lot without taking piracetam.

In any case, piracetam either has no effect on me in the reasoning domains I'm interested in, or else its effect is so small that it is overwhelmed even by relatively minor fatigue effects.

Comment author: Jack 28 September 2011 05:33:02PM *  2 points [-]

In this case the fact that many people believe in a Native American-Israelite connection is not evidence against a Native American-Israelite connection, just evidence against the claims of Mormon archaeologists on the subject. You might update similarly if you learned that particular archaeologist was incompetent or dishonest. The causal connection between the fact and the evidence has been undermined. But I think this is more a matter of formalizing how we represent evidence, I doubt there is a substantive disagreement over what heuristics are helpful. Semantics, I think.

Comment author: Yvain 28 September 2011 07:23:06PM 0 points [-]

I agree. Although Mormonism may not be a good example because only certain people are Mormons. If we use an example that potentially casts doubt on all humans (for example, conspiracy theories), then it not only means that all previous evidence you've heard from others becomes less believable, but even that your own calculations on the subject are now suspect since you are probably subject to the same bias as everyone else.

Reversed stupidity sometimes provides useful information

16 Yvain 28 September 2011 10:28AM

In his recent CATO article, Reversed Stupidity Is Not Intelligence, Eliezer writes:

To psychoanalyze these people’s flaws, even correctly, and even if they constitute a numerical majority of the people talking about “quantum,” says nothing at all about whether the smartest people who believe in “quantum” might perhaps be justified in doing so  ...  there are large numbers of embarrassing people who believe in flying saucers, but this cannot possibly be Bayesian evidence against the presence of aliens, unless you believe that aliens would suppress flying-saucer cults, so that we are less likely to see flying-saucer cults if aliens exist than if they do not exist. So even if you have truly and correctly identified a cluster of people who believe X for very bad, no good, awful, non-virtuous reasons, one does not properly conclude not-X, but rather calls it all not-evidence.

I think the statement makes a correct point - don't dismiss an idea just because a few proponents are stupid - but is too strong as written. In some cases, we can derive information about the truth of a proposition by psychoanalyzing reasons for believing it.

There are certain propositions that people are likely to assert regardless of whether or not they are true. Maybe they're useful for status disputes, or part of a community membership test, or just synchronize well with particula human biases. "X proves the existence of God" commonly gets asserted whether or not X actually proves the existence of God. Anything that supports one race, gender, political party, or ideology over another is also suspect. Let's call these sorts of propositions "popular claims". Some true propositions might be popular claims, but popular claims are popular whether or not they are true.

Some popular claims are surprising. Without knowing anything about modern society, one might not predict that diluting chemicals thousands of times to cure diseases, or claiming the government is hiding alien bodies, would be common failure modes. You don't know these are popular claims until you hear them.

If a very large group of people make a certain assertion, and you always find it to be false, you now have very good evidence that it's a popular claim, a proposition that people will very often assert even if it's false.

Normally, when someone asserts a proposition, you assume they have good evidence for it - in Bayesian terms, the probability that they would assert it is higher if there is evidence than if there is not evidence. But people assert popular claims very often even when there is no evidence for them, so someone asserting a popular claim provides no (or little) evidence for it, leaving you back at whatever its prior is.

Time for an example: suppose two respected archaeologists (who happen to be Mormon) publish two papers on the same day. The first archaeologist claims to have found evidence that Native Americans are descended from ancient Israelites. The second archaeologist claims to have found evidence that Zulus are descended from Australian aborigines.

On the face of it, these two claims are about equally crazy-sounding. But I would be much more likely to pay attention to the claim that the Zulus are descended from aborigines. I know that the Mormons have a bias in favor of believing Indians are descended from Israelites, and probably whatever new evidence the archaeologist thinks she's found was just motivated by this same bias. But no one believes Zulus are descended from Australians. If someone claims they are, she must have some new and interesting reason to think so.

(to put it another way, we expect a Mormon to privilege the hypothesis of Israelite descent; her religion has already picked it out of hypothesis-space. We don't expect a Mormon to privilege the hypothesis of Australian descent, so it's more likely that she came to it honestly).

If then I were to learn that there was a large community of Mormons who interpreted their scripture to say that Zulus were descended from Australians, I would consider it much more likely that the second archaeologist was also just parroting a religious bias, and I would no longer be quite as interested in reading her paper.

In this case, reversed stupidity is intelligence - learning that many people believed in an Australian-Zulu connection for religious reasons decreases my probability that the posited Australian-Zulu connection is real. I can never go lower than my whatever my prior for an Australian - Zulu connection would be, but I can discount a lot of the evidence that might otherwise take me above my prior.

So in summary, a proposition asserted for stupid reasons can raise your probability that it is the sort of proposition that people assert for stupid reasons, which in turn can lower your probability that the next person to assert it will have smart reasons for doing so. Reversed stupidity can never bring the probability of an idea lower than its prior, but it can help you discount evidence that would otherwise bring it higher.

Comment author: Yvain 28 September 2011 09:30:05AM *  25 points [-]

I always find it a red flag when it seems like an entire group of highly-educated people is doing something ridiculously stupid. If assuming the brain thinks in terms of necessary-and-sufficient would be really stupid, maybe that's not what conceptual analysts are doing.

The idea that our brain's fuzzy type-1 thinking can be translated into precise type-2 thinking is one of the foundations of science and mathematics, not to mention philosophy. I'd been drawing and seeing circles for years as a child before I learned that they were the 2-D set of points equidistant from a center point, but this latter definition accurately captures a necessary and sufficient condition for circles. Anyone who says "your brain doesn't really process circles based on that definition, it's just pattern-matching other circles you've seen" would be missing the point.

And this process sometimes works even with natural categories. Wikipedia defines "birds" as "feathered, winged, bipedal, endothermic (warm-blooded), egg-laying, vertebrate animals", and as far as I can tell, this is necessary and sufficient for birds (some sources say kiwis are wingless, but others say they have small, vestigial wings). Although birds are the classic example of a fuzzy mental category, like the "circle" category it turns out to have a pretty good necessary-and-sufficient definition after all. Likewise, "fish" are "all gill-bearing aquatic vertebrate animals that lack limbs with digits".

Even if the above turn out to be incorrect (we find an animal we intuitively classify as a fish that does not meet that definition), it's still interesting and potentially useful to have a prediction rule that works 99+% of the time. And someone who started out believing whales to be fish could, armed with such a 99%-rule, correct her error.

So even if our brains don't naturally think in terms of necessary-and-sufficient, it's not immediately obvious that it's stupid and impossible to try to come up with necessary-and-sufficient conditions for our categories.

There may be some sets of borders in thingspace which are better than others, in the same way that there are some borders for an independent Palestinian state that are better than others (even though we're not sure exactly where the border should be, sticking Tel Aviv in Palestine, or Ramallah in Israel, would be a mistake).

Fixing borders in thingspace can determine the status of edge cases. Sometimes this can even be useful; for example, if I am allergic to fish, then having a correct boundary for "fish" will let me know I can safely eat whale meat. I may not know exactly what chemical in fish causes my allergic reaction, or even know that allergy is an immune reaction to specific chemicals - but being able to draw the category boundaries accurately will "miraculously" predict that whale meat will not trigger my allergy.

Or more realistically, coming up with a set of criteria for "good" will help me determine that stoning homosexuals is bad, even if I previously didn't realize this. And philosophers have come up with some pretty good definitions for "good" that can do this - not universally accepted, by any means, but useful to those who know them.

I think you have a strong case against conceptual analysis, and some very intelligent commentary on where conceptual analysis does work - but it's all in Conceptual Analysis and Moral Theory. This post seems to be attacking a straw man by accusing conceptual analysts of necessarily trying to model the human brain and then doing a bad job of it.

Comment author: Yvain 27 September 2011 09:05:31AM 4 points [-]

I don't know how important it will be for a naive sample, but if the sample includes transhumanists, you might need to clarify your definitions of "humanity" and "extinct". If we transition to beings of pure energy that bear no more relation to humans than humans do to cockroaches, is that "humanity becomes extinct" or not?

Comment author: steven0461 25 September 2011 10:09:32PM 7 points [-]

I consider it morally important to make attempting to profiteer off of other people's scientific research as unprofitable and unpleasant as possible

Are you the same Yvain who wrote that consequentialism FAQ and that optimal philanthropy article? Surely the lesson from those topics is that it's not morally important to make your own life more difficult in service of "good causes" that are actually relatively unimportant.

Comment author: Yvain 26 September 2011 05:33:46AM *  5 points [-]

By "morally important", I didn't mean "this is the most important moral issue", only "something that moral considerations should bear upon". "Morally charged" might be more accurate.

It's always going to be "irrational" to punish people, but "super-rational" considerations say that going to disproportionate lengths to punish people makes people less likely to cause trouble. So I admit some of my beliefs here are disproportionate, but I'm okay with that.

But I do think that if gated journals retard the advance of science by, say, 2% (and there's some reason to think they affect real researchers and not just amateurs), that's not trivially unimportant.

Of course, if a gated journal article is the only thing between your research and a cure for cancer, you pay them the money.

Comment author: Yvain 25 September 2011 08:10:14PM 8 points [-]

Can you point out the particular paragraph in your neuroscience post you're referring to? I looked and couldn't find anything that obviously fit the bill.

I would be very worried that you're describing something closer to "wanting", but that CEV should take as input something closer to "approving"

Comment author: Yvain 24 September 2011 05:51:28PM *  16 points [-]

Nothing says "We are high-status members of mainstream society and not just a bunch of out-of-touch geeks" like a Segway!

Comment author: Yvain 21 September 2011 06:52:58PM *  19 points [-]

I appreciate and agree with the principle behind this post, but when a store wants to charge me for using the bathroom I either find a friendlier store or else I hand them the money with a smile and never buy anything from that store ever again.

There are certainly sources of knowledge that are not cheap to produce and which deserve our funding and our appreciation. But I am not going to give eg gated journals one cent more than I am absolutely forced to, and I consider it morally important to make attempting to profiteer off of other people's scientific research as unprofitable and unpleasant as possible.

Comment author: wedrifid 21 September 2011 01:30:18PM -1 points [-]

Let's define our future selves as agents that we can strongly influence, and that we strongly care about.

This predicts our children are to some degree our future selves.

Predictions seem to be a different thing in nature to definitions. The definition is terrible but it, well, by definition doesn't make a prediction.

Comment author: Yvain 21 September 2011 06:42:26PM *  7 points [-]

Should I have used a different word? Probably! But I will now proceed to a complex justification of my word choice anyway!

A lot of philosophy seems to be coming up with explicit definitions that fit our implicit mental categories - see Luke's post on conceptual analysis (which I might be misunderstanding). Part of this project is the hope that our implicit mental categories are genuinely based off, or correspond to, an explicit algorithmizable definition. For example, one facet of utilitarianism is the hope that the principle of utility is a legitimate algorithmization of our fuzzy mental concept of "moral".

This kind of philosophy usually ends up in a give-and-take, where for example Plato defines Man as a featherless biped, and Diogenes says that a plucked chicken meets the definition. Part of what Diogenes is doing is saying that if Plato's definition were identical to our implicit mental category, we would implicitly common-sensically identify a chicken as human. But we implicitly common-sensically recognize a chicken is not human, therefore our minds cannot be working off the definition "featherless biped".

This is the link between defining and predicting. Plato has proposed a theory, that when the mind evaluates humanity, it uses a featherless-biped detector. Diogenes is pointing out Plato's theory makes a false prediction: that people implicitly recognize chickens as humans. This disproves Plato's theory, and so the definition is wrong.

I suppose this must be my mental concept of what we're doing when defining a term like "self", which is what impels me to use "define" and "predict" in similar ways.

Comment author: Yvain 21 September 2011 11:58:17AM *  6 points [-]

Let's define our future selves as agents that we can strongly influence, and that we strongly care about.

This predicts our children are to some degree our future selves. I'm not sure if that's a plus or a minus for this theory.

I don't think there's any metaphysical meaning to "X is the same person as Y", but our mental programs take it as a background assumption that we're the same as our future selves for the obvious evolutionary reasons. Identity with our future selves is on the bottom level of human values, like preferring pleasure to pain, and there's no need to justify it further.

I don't know if this is the same as your theory or not.

Comment author: MinibearRex 20 September 2011 09:04:41PM 7 points [-]

Is this really a rationality quote, is it just pro-Yudkowsky?

It does set a standard for the clarity of any writing you do, but I've seen substantially better quotes on that topic before.

Comment author: Yvain 21 September 2011 10:02:15AM *  2 points [-]

Related to hindsight bias and inferential distances. I'd sort of noticed this happening before, but if I hadn't realized other people had the same experience I probably would have underestimated the degree to which rationality had changed my worldview and so underestimated the positive effect of spreading it to others.

Comment author: Yvain 19 September 2011 06:41:59PM 10 points [-]

I recently found an interesting study that bears on the doctor example. Christensen and Bushyhead (1981) find that when asked to make clinical judgments, doctors usually take base rates into account quite accurately, even though when they are asked to explicitly do statistical problems involving base rates they usually get them wrong.

Comment author: Yvain 19 September 2011 06:22:15PM 11 points [-]

I think there's a few posts by Yudkowsky that I think deserve the highest praise one can give to a philosopher's writing: That, on rereading them, I have no idea what I found so mindblowing about them the first time. Everything they say seems patently obvious now!

-- Ari Rahikkala

Comment author: PhilGoetz 15 September 2011 10:28:29PM *  -2 points [-]

Right now, Virginia is regulating abortion clinics, making them meet hospital standards, in order to protect womens' safety. Yet I don't think there are any known cases of failures from too little regulation in Virginia abortion clinics. At least, I haven't heard any brought up.

Comment author: Yvain 16 September 2011 08:19:56AM 1 point [-]

Do you mean no cases of failure from too little regulation that couldn't have been solved more cleverly by nonregulatory means, no cases of failure where the "solving" regulation didn't have problems of its own, or just that you literally can't think of any any cases ever of failures from too little regulation?

If you really mean the last, then avoiding modern-day issues so we don't get into a fight, the Triangle Shirtwaist Factory, thalidomide babies, and the Exxon Valdez oil spill seem like go-to historical examples.

Comment author: khafra 15 September 2011 05:02:27PM 5 points [-]

Somebody mentioned Aleister Crowley's quotes on LW a little while ago; so:

There seems to be much misunderstanding about True Will ... The fact of a person being a gentleman is as much an ineluctable factor as any possible spiritual experience; in fact, it is possible, even probable, that a man may be misled by the enthusiasm of an illumination, and if he should find apparent conflict between his spiritual duty and his duty to honour, it is almost sure evidence that a trap is being laid for him and he should unhesitatingly stick to the course which ordinary decency indicates ... I wish to say definitely, once and for all, that people who do not understand and accept this position have utterly failed to grasp the fundamental principles of the Law of Thelema.

-- Magical Diaries of Aleister Crowley : Tunisia 1923 (1996), edited by Stephen Skinner p.21

Comment author: Yvain 15 September 2011 05:24:29PM *  9 points [-]

If one is skeptical of the existence of Thelema or of the validity of these spiritual experiences, then this sounds a lot like religious leaders who say "Sure, believe in Heaven. But don't commit suicide to get there faster. Or commit homicide to get other people there faster. Or do anything else that contradicts ordinary decency."

Part of the fun of being right is that when your system contradicts ordinary decency, you get to at least consider siding with your system.

(although hopefully if your system is right you will choose not to, for the right reasons.)

Comment author: Yvain 15 September 2011 05:22:53AM 10 points [-]

You could attempt to adopt a strategy of always following your commitments. From your current perspective this is useful but once you have learned your strategy has failed, what's to prevent you from just disregarding the strategy?

Disregarding it once will convince yourself and others that you will disregard it in the future, and remove your ability to make other precommitments.

The nuclear war example is more complicated, because presumably having a nuclear war will be the last thing you ever do. I would credit it to evolved instincts. Evolution "knows" that precommitments are important, so it gives us the desire to follow them even when it is not immediately rational to do so - for example, a lust for revenge that ought to be sufficient to make us retaliate in nuclear war, or a concept of "honor" that does the same.

Comment author: [deleted] 09 September 2011 12:41:15PM 6 points [-]

Maybe it's antisocial to keep asking these sorts of questions, I hope not. Do either of you have any idea where I can find a less-wrongian friendly description of this "hard problem". Everything I've read that tries to describe it in the past is full of snippets like "something that it's like to be" and "subjective qualitative" and other such things I have no ability to understand...

Comment author: Yvain 10 September 2011 03:13:18PM 3 points [-]

It's kind of supposed to be hard to explain, but...hmmmm...maybe something like "why is there a subjectively perceived difference between sleepwalking through your life and being awake?"

If we imagine a sort of "perfect sleepwalker" who, while sleepwalking, could hold conversations, go to work, write poetry, and do anything else that people do while awake exactly as waking people do it - even to the point where if we ask her "Are you awake?" she answers "Yes." - then it might be necessarily impossible for us outsiders to distinguish her sleep from her waking.

But we feel an intuitive believe that she should be able to do so easily. If she's awake, she can notice her awakeness and all the sensations she's feeling and experiences she's having. If she's sleeping, then it doesn't even make sense to "experience" not being awake, because there's no one "at home" to do the experiencing.

An equivalent interpretation of the problem revolves around qualia. Suppose that your experience of "red" was everyone else's experience of "blue". You would never be able to confirm this by talking to other people - you would say things like "blue is the color of the sky and the sea and short-wavelength light" and they would agree with you, but you would be thinking of red when you said it, and everyone else would be thinking of blue. This "experience of blue" which is separate from statements about blue or concepts surrounding blue is the "quale" (plural "qualia") of blue.

Intuition tells us one difference between the sleepwalker and the awake person is that if you ask the sleepwalker what color a stop sign is, the light rays would hit her eyes, go through a chain of neurons in her brain, and produce the response "it's red". The same thing would happen in the awake person, but she'd also have the conscious visual experience qualia thing where she "sees" a certain color in her "mind's eye".

The hard problem is whether there's a difference between awake people with qualia and perfect sleepwalkers (aka "p-zombies") without qualia, and, if so, what causes that difference.

In response to comment by Yvain on Pain
Comment author: lessdazed 09 September 2011 05:48:47PM 1 point [-]

We pretty much agree on the issue itself. I don't see why a person gets to "own" their pain, someone's pain can be disutility for a second person who cares about it.

I agree with kpreid that you are wrong about what others are saying, that's mostly it.

In response to comment by lessdazed on Pain
Comment author: Yvain 09 September 2011 08:28:39PM 1 point [-]

Okay, I assume it's a misunderstanding on my part and sorry about that. (lays dead thread to rest)

Comment author: Matt_Simpson 08 September 2011 03:25:11PM 2 points [-]

Yes, but you've merely labeled the tendency. You haven't explained it. I can spin some plausible sounding evolutionary explanations, but what I'm really interested in is the level of cognitive science.

Comment author: Yvain 09 September 2011 12:23:02PM 7 points [-]

If I had to come up with a cognitive science explanation for pro-regulation, regardless of political considerations around whether regulation was good or bad, it would be that failures from too little regulation are obvious, direct, and heartrending (child dies of toxic unproven medicine) and failures from too much regulation are distributed and invisible (child dies of cancer, with no one knowing that a cure sits in a lab somewhere but it's too expensive to license it).

This is because regulation, as a specific action taken to stop a problem, gets to optimize for fighting the most obvious, scary problems in the most direct way - whereas nonregulation, as a null action, doesn't get to optimize for that at all.

Comment author: Yvain 09 September 2011 11:55:07AM 8 points [-]

Aren't most of the people who say consciousness is a mystery talking about the hard problem, whereas global-workspace theory and higher-order theory and the like address the easy problem?

In response to comment by Yvain on Pain
Comment author: lessdazed 08 September 2011 11:04:35PM 1 point [-]

I try to make space for people to recant old positions because I certainly need it.

You're saying the points are implied by the first comment in the thread, and I don't think they are. I see by your clarifications that I agree with you significantly about the issue itself but I think you are very wrong about the implications of "Pain...is bad, because it interferes with working towards what one values."

For instance:

Consider someone whose only ambition is to collect every Pokemon in the world. Kpreid's scenario suggests a dichotomy: either it is okay to cause this person pain, or the only reason not to cause this person pain is because it might prevent Pokemons from being collected.

If my goal is for people to not be in pain, pain to that collector is obviously bad. If my goal is for people who don't want to be in pain not to be in pain, a consequentialist calculation probably indicates I should still work to minimize the pain of people who protest that they don't care despite their statements.

In response to comment by lessdazed on Pain
Comment author: Yvain 09 September 2011 11:51:43AM 0 points [-]

I don't disagree that "pain [can be] bad because it interferes with working toward what one values", I only disagree that that is the only reason pain can possibly be bad.

Maybe the confusion here is translating between pain and utility. I view KPReid as making the claim:

"Pain in itself should not be considered disutility. Only failure to achieve a goal should be considered disutility, and pain should be counted as decreasing utility only insofar as it affects that."

(where 'goal' here is an explicit goal like 'collect Pokemon' and not an implicit goal like 'avoid pain'. If all kpreid was trying to say was that "avoid pain" can be considered a "goal", I agree. In the Pokemon example, I'm assuming a neurotypical Pokemon collector who may have dedicated her life to collecting Pokemon, but still feels pain in the same way everyone else does and dislikes it - not a nonhuman Pokemon-maximizer)

I consider myself as making the different claim:

"Pain in itself can be disutility if the person involved does not want pain."

Note that under my interpretation, it doesn't matter whether or not the pain conveys information; information may be a counterbalancing factor that outweighs the disutility of the pain, but the pain is still bad. See my response to Silas.

I'm still not convinced we don't mostly agree on this issue.

In response to comment by Yvain on Pain
Comment author: lessdazed 08 September 2011 09:59:10PM 1 point [-]

Well, you're usually right about everything, so this is quite a break in the pattern. ;-)

In response to comment by lessdazed on Pain
Comment author: Yvain 08 September 2011 10:10:41PM 3 points [-]

That's the most confusing way of being disagreed with I've ever experienced :)

...you are aware that I'm attacking each of the bullet points in the comment above, not agreeing with them - right?

Comment author: Matt_Simpson 07 September 2011 09:11:23PM *  18 points [-]

From the beginning of Eliezer's piece

I think my libertarianism rests chiefly on the empirical proposition—a factual belief which is either false or true, depending on how the universe actually works—that 90% of the time you have a bright idea like “offer government mortgage guarantees so that more people can own houses,” someone will somehow manage to screw it up, or there’ll be side effects you didn’t think about, and most of the time you’ll end up doing more harm than good, and the next time won’t be much different from the last time.

I think there's an interesting congitive phenomenon going on here. If you tell the average Joe that you propose to deregulate some industry or activity, they easily call to mind the potential negative consequences - "they'll just pollute as much as they want now" or "only the biggest companies will buy up the fishing quotas" etc. What the average Joe doesn't think about are the potential negative effects of regulation - unintended consequences and regulatory capture for example. Regulatory capture is a sort of funny case because though the concept was invented by a Marxist (if memory serves), only economists and libertarians seem to think about it. When the average Joe does recognize these shortcomings, they virtually always get to designing a "better" law, agency, and/or policy. For evidence, I offer r/politics, though there's an admittedly liberal slant there (relative to the population).

I'm not sure what is causing this bias or even how to categorize it, but it seems related to how most people respond to the broken window fallacy. Some effects are "seen" and others "unseen", or rather more easily seen and less easily seen (which correspond to something like direct effects and indirect effects), and the average person is much better at anticipating "seen" effects. Something like this is going on in the political case - at least in terms of unintended consequences, but that's not all of it. Regulatory capture occurs, in part, because people in the government are no more or less corrupt than people outside of the government. Most people are quick to suspect private companies of being corrupt, but never a new regulatory agency to reign in that private company. The solution to a government failure is always government. Rarely does the average Joe even ask the question "is the government even good at doing this sort of thing?" At least at the surface level, this looks nothing like "seen" and "unseen." I'm not entirely sure what's going on here, but I suspect figuring it out will help explain political preferences.

Comment author: Yvain 08 September 2011 08:53:49PM *  20 points [-]

At the obvious risk of making this too political (in my defense, everyone else started it), I'm not sure "no one except us ever realizes how bad regulation is" is the real problem here.

In my metacontrarianism post I talked about how when someone discovers a counterintuitive idea that only smart people can understand, they can become very excited and use it as a membership test for the Secret Society Of Smart People, to the point where they enter a Happy Death Spiral and blow it out of all proportion.

(for example, it's easy to think of ways modern Western civilization is better than primitive tribal civilization - more wealth, less prejudice, fewer tapeworms. There are also a few counterintuitive hard-to-understand ways primitive tribal civilizations are better than us - closer family bonds, more attuned to the natural world. And so many intellectuals focus on these to the exclusion of everything else and talk about how Western society is hopelessly corrupt and evil. Where they should be saying "Western civ is great, but not quite as great as the average person might think," we instead get "Western civ is awful." This passes unchallenged because the good things about Western civ like absence of tapeworms have become background noise to which everyone has adjusted. One interesting marker for this kind of behavior is that people still feel contrarian when they say it even though everyone who hears it agrees with them.)

I feel that some aspects of opposition to regulation might be this kind of behavior. There's a large background of successful regulation (like banning lead and fluorinating water and not dumping trash on the streets and so on) that no one ever notices (unless they've just come back from a country that doesn't have it!) There's also some highly available examples of regulation that goes wrong. People feel like these must be hard to catch and so can start a death spiral around how intelligent they are to notice.

So my explanation for why so many people don't understand that regulation is bad would be that this lack of understanding is not an actual phenomenon that exists in the real world. The US House of Representatives is currently dominated by a movement entirely based on eliminating as many regulations as possible as loudly as possible, and even their opponents pay lip service to the same ideas. This is another one of those cases where people say something everyone agrees with but still consider themselves "contrarian", which as I mentioned above makes me suspicious of a signaling effort.

I think the problem is less that no one understands regulation can be bad, than that talking about how no one understands regulation can be bad is a powerful signal.

...which still leaves the question of why most people who assert that they hate regulation support most actual regulation (eg Ron Paul will never get elected). I can think of a few explanations. Most cynically, they back away from their signaling when it has consequences, the same way the anti-Western-civ people would never actually move to a primitive tribe. Less cynically, there may be a status quo bias in which they process "new" regulation differently from existing regulation (though that wouldn't explain why people tend to support a lot of new regulation too). I think most plausibly it's a difference between Near and Far mode reasoning - regulation in the abstract is Far, any specific law is Near - and neither mode is necessarily "better" than the other.

In response to comment by Yvain on Pain
Comment author: lessdazed 06 September 2011 08:46:05PM 1 point [-]

I'm curious as to whether or not you still stand by the opinions and reasoning expressed in this comment.

In response to comment by lessdazed on Pain
Comment author: Yvain 08 September 2011 08:28:04PM 0 points [-]

Yes, and by the clarifications elsewhere on this thread. Is there some reason I shouldn't?

Comment author: r_claypool 07 September 2011 04:15:01AM *  4 points [-]

Lydia McGrew responded to you saying:

... the earlier commentator who says that the probability is "approximately 1" that there would be made-up resurrection stories (and apparently thinks that this applies to the gospels) ignores various obvious distinctions. For example, the distinction between stories by people who had nothing to gain and everything to lose for making up such stories and people who had nothing to lose and something to gain by doing so. Also, the distinction between people's elaborating stories when they themselves were in a position to know what really happened and people who were not in a position to know what really happened.

We are talking in the paper about what the disciples themselves claimed. They were in a position to know whether what they were claiming was true or false, and they had a great deal to lose and nothing to gain by simply making up such tales. - link

Comment author: Yvain 08 September 2011 08:26:30PM 6 points [-]

Thanks for pointing that out. Needless to say I don't agree, but I respect her decision not to get in an endless internet argument about it.

Comment author: Yvain 03 September 2011 02:01:11PM *  12 points [-]

A strange result. When I hear "spatial abilities depend on nurture", I expect something like "our society teaches boys to be good at sports and video games and other things that require spatial reasoning, so they have more practice."

This suggests it's a function of social status, which raises the question of why social status increases spatial reasoning skills. The authors admit education only makes up a third of the difference (and if education was the only issue, we would expect language skills to suffer in the same way, but the "popular wisdom" is that women usually test for better language skills). Homeownership is another weird one - why should owning a home give you better spatial skills (if we assume that the homeowner and their spouse both navigate the rooms of the house the same amount and so on).

The only explanation I can think of is stereotype threat, but again that makes the whole "women are better at language skills" thing weird if we can't explain where the stereotypes came from originally. Now I want to know whether the stereotype that men are better at spatial and women better at language skills evolved multiple times in multiple societies, or whether it's just a function of Westerners introducing it everywhere they went.

Comment author: lukeprog 30 August 2011 04:42:29AM *  6 points [-]

Nice little article.

Comment author: Yvain 02 September 2011 05:25:11PM 10 points [-]

This explains a lot.

Comment author: Eliezer_Yudkowsky 02 September 2011 09:57:56AM 17 points [-]

<mental model of Michael Vassar says>This strikes me as a nerdism. If you don't find less intelligent people easier to manipulate, you must be working on sympathetic models of them instead of causal ones. I expect that experience would cure this, and after a few months of empirical practice and updating on the task of reasoning with fools, you would find it was actually easier to get them to do whatever you wanted - if you could manage to actually try a lot of different things and notice what worked, instead of being incredulous and indignant at their apparent reasoning errors.</Vassar>

Comment author: Yvain 02 September 2011 10:36:34AM 14 points [-]

Upvoted the original for reference to Prince of Nothing series. And upvoted this comment for the terms "sympathetic model" and "causal model", which is one of those times that having the right word for a concept you've been trying to understand is worth a month of trying to untangle things in your head.

...although now I'm not sure whether I should upvote Eliezer or Michael Vassar. It seems kind of unfair to deny Michael an upvote just because the specific instantiation of his algorithm that said this happened to be running on Eliezer's brain at the time.

Comment author: Yvain 02 September 2011 10:10:29AM *  47 points [-]

I've been watching with interest the debates around how good minicamp was. I think we need to distinguish between two hypotheses:

A) Minicamp was well-run, the participants enjoyed it and subjectively estimated it was helpful, and it made everyone involved much more enthusiastic about rationality and motivated to pursue positive self-change.

B) Minicamp had objective effects on measurable rationality parameters like calibration, and objective long-term effects on things like lifetime success, tendency to help rationality-related causes, and ability of participants to enjoy their life.

Everyone who's talking about how obvious it is that minicamp was a stupendous success is talking about A, and everyone who's saying they're not convinced is talking about B.

Most self-help doesn't work - so with zero background information about a camp, our prior probability of B is low. That this is a rationalist camp is extra information: P(B|Rationality_Camp) is greater than P(B) alone if we believe rationality is a more effective self-help strategy than average. But this just brings us to where we were before the camp started; to say the evidence in the post above increases that estimate we've got to investigate P (B|A) - the probability that, given a camp gets glowing reviews and everyone loved it and thinks it changed their life, the it really is effective.

But A is a common feature of almost all self-help camps - googling "Christian retreat testimonial" can be very enlightening (add the phrase 'changed my life' to the query for best results). I think most rationalists would be very skeptical of most of the camps that manage to get such glowing reviews from their participants. So P(B|A) - the probability that data shows a real long-term effect given that everyone loved it and is wildly enthusiastic - may not be much higher than P(B).

So if you're trying to prove A - that the camp was successful and everyone loved it and felt very motivated - you've more than succeeded by now. If you're trying to prove B, keeping on giving more and more evidence for A isn't really the way to go.

A better suggestion might be to tell people "We have strong evidence you'll love the camp and feel transformed and enlightened, and we have some evidence that it will help because rationality is teachable and we're trying to gather more specific evidence as the program continues."

Comment author: Yvain 01 September 2011 05:08:02PM 7 points [-]

I have data from an unpublished experiment on factors affecting calibration. People with higher education levels were better calibrated, and people from more "rational" occupations like "scientist" or "computer programmer" were better calibrated than people from less "rational" occupations like sales or design. People whose job involved working with risk and probabilities directly (eg investment banker, weatherman) were just below scientists.

There was an attempt to investigate whether gambling helped, but it got contradictory results: frequent gamblers were worse calibrated on questions like "What percentage chance France is bigger than Germany?" but better calibrated at predicting future events. I don't understand why this happened.

This doesn't distinguish between education and science education making people more rational, and more rational people getting more education and going into science, but it's at least a little positive.

In response to Rational Home Buying
Comment author: MarkusRamikin 27 August 2011 07:15:45AM *  3 points [-]

Great article.

In a telling experiment under the same protocol as the ones listed above, people asked to reflect upon their choices were more likely to choose the house with the extra room for Grandma than the house with the shorter commute times, because the extra reflection gave more opportunity for the availability heuristic to come into play.

I am curious about the part after "because". Did the experiment you're referring to actually distinguish that explanation (availability heuristic) from other possible ones (such as signalling value, greater ease of putting some factors into words, etc)?

One of the reasons I'm curious is because I'd have thought that availability heuristic affects one more strongly when one takes a decision quickly and with little reflection.

Comment author: Yvain 01 September 2011 11:17:22AM 0 points [-]

I've only read a distillation of the experiments (which had the same "because" I used) but not the papers themselves. The distillation mentioned that the scientists conducted interviews with the participants (whatever that's worth) and that this was their interpretation of a broad category of results from several similar experiments. I don't know if there were actual efforts to exclude signaling etc. See Lehrer, How We Decide, Ch. 5 for more detail.

Comment author: orthonormal 31 August 2011 01:59:25PM *  7 points [-]

Good point. My first objection is the same as prase's, my second is that a government that depends on popular support shouldn't enforce policies that creep out the citizens (either because they'd lose the next election in a landslide to a party with different values or decision theory, or because in a Parfit's Hitchhiker way, if it were clear they would do such things then they'd have lost the previous election).

My third is that the creepiness here comes from our very real (and very understandable in consequentialist terms) fear of allowing the government too much power over citizens' life and death. If instead you asked, should we make it legal for people to voluntarily off themselves when by so doing they can save several other lives, and should we make it an honorable thing to do, then I'm not nearly as disturbed by the idea. (There are myriad variations to try out, but generally whenever it gets creepy I can identify some sort of awful incentive structure being set up.)

Comment author: Yvain 31 August 2011 05:43:26PM 1 point [-]

I probably shouldn't have said "government". We get the same issues if the doctor just wanders around town, spots a genetically compatible person with his Super-Doctor-Vision, and kills them at their house. Or really through any means other than waiting for compatible donors to show up at the hospital.

Your point five is well taken, though.

Comment author: Yvain 31 August 2011 08:38:29AM 12 points [-]

What if, instead of deciding whether the doctor murders the patient in secret when she comes to the hospital, we have to decide whether the government (perhaps armed with genetic screening results seized from a police databases and companies like 23andMe) passes a law allowing police to openly kill and confiscate organs from anyone whose organs could presumably save five or more transplant patients?

As far as I can tell, this would have no bad effects beyond the obvious one of killing the people involved - it wouldn't make people less likely to go to hospitals or anything - but it keeps most of the creepiness of the original. Which makes me think although everything you say in this post is both true and important (and I've upvoted it) it doesn't get to the heart of why most people are creeped out by the transplant example.

Comment author: Yvain 31 August 2011 07:21:56AM 14 points [-]

Agree that the article has serious structure and style problems, but a quick check at Wikipedia confirms that DI seems to be approximately as awesome as you say it is, and the biases of the educational establishment certainly seem worth investigating. Upvoted.

Comment author: Yvain 29 August 2011 03:42:08PM 6 points [-]

I had good results with The Moral Animal (a good intro to evo psych). It might be easier to challenge some of the foundations upon which theism rests and let them draw their own conclusions than to attack theism directly and activate defense mechanisms.

Comment author: Yvain 29 August 2011 12:41:56PM 3 points [-]

Excellent article and reminds me how little I really know about Bayesian techniques and statistics.

Maybe when you're done grounding morality and unraveling the human brain and developing friendly AI and disproving the existence of God, you can write a series of posts picking up where "Intuitive Explanation of Bayes Theorem" and "Technical Explanation of Technical Explanation" left off.

Comment author: ArisKatsaris 27 August 2011 12:06:00AM *  6 points [-]

I'd like to warn of a slight ambiguity in the stated commitment Eliezer made in this latest chapter. I'm especially worried that since money are involved, some people may feel cheated if the pledge isn't clarified ASAP and it ends up being the weaker form of the pledge.

"I will release completed chapters every X days" -- does it mean that the chapters that are currently completed, will be released one every X days? How many chapters are those? Or does it mean Eliezer pledges to complete and release a chapter on such a schedule for the remaining duration of HP:MoR?

If it's the stronger form of the pledge, I doubt anyone will complain -- though I'm a bit skeptical that it's wise to make a promise as hard as "a chapter every three days".

Comment author: Yvain 27 August 2011 12:07:54PM *  16 points [-]

<whiny nitpicking complaint>

I donated, because it was a good reminder that I should be donating anyway, but the reward kind of leaves me cold. If Eliezer writes new chapters after 78 at a constant rate while publishing existing chapters, then if he publishes all his existing chapters really quickly he won't get many new chapters done in the interim and we'll have to wait a longer time after 78 for 79. If he publishes existing chapters slowly, he'll have 79 done by the time he gets to 78, and can publish those at a steady rate while working on 80+.

So instead of donating to get more total HPatMOR, donations change the pacing away from "slow and steady" and toward "quick burst of chapters now, then a longer interval", with the end date of the fic not changing at all. Personally, I would prefer the steadier pacing.

</whiny nitpicking complaint>

In response to Rational Home Buying
Comment author: gjm 26 August 2011 09:06:13PM 8 points [-]

a neighborhood ... where you'll be relatively high on the social ladder.

That seems to be the reverse of what the research you cited indicates (rich people in rich neighbourhoors of poor counties in America are happy). Don't you want to be in a rich neighbourhood (so you're surrounded by nice things and reminders of your high status) in a poorer region (so you have something worse to compare against) of a rich country (for all the infrastructural benefits that brings)?

In response to comment by gjm on Rational Home Buying
Comment author: Yvain 27 August 2011 09:41:05AM 1 point [-]

I've changed the word "neighborhood" to "area" here, but this was actually the point I was least certain about. A lot of the theory and some studies I see mentioned but can't find do tend to suggest neighborhood, but the Firebaugh study, which is the best I could find an abstract for says not neighborhood but region.

I don't know to what degree that contradicts the earlier results, and since I can't even find full text for Firebaugh, I'm not sure whether it means anything more profound than "try to avoid a house in the middle of a loud dirty ghetto, because you wouldn't like that."

I'm also wary of this whole area because of the controversy around the Easterlin Paradox

In response to Rational Home Buying
Comment author: twanvl 26 August 2011 09:29:01PM 12 points [-]

it costs $20 at a nearby store, and it costs $40 at a store that's fifteen minutes away ... which costs $2000 at a nearby store, and $2020 at a store that's fifteen minutes

Shouldn't these numbers be the other way around? I.e $40,$20 and $2020 and $2000? Why would you drive fifteen minutes to pay an extra $20?

Comment author: Yvain 27 August 2011 09:28:52AM 2 points [-]

Thanks, fixed.

In response to Rational Home Buying
Comment author: gwern 27 August 2011 12:41:09AM 9 points [-]

You don't have to live in the Amazon to get a benefit: even children in a concrete building with a tiny "green island" boasting a single tree did better than their peers in a building without such an island.

The citation seems to be missing for this one.

In response to comment by gwern on Rational Home Buying
Comment author: Yvain 27 August 2011 09:25:02AM *  5 points [-]

Sorry about that:

Faber Taylor, A., Kuo, F.E., & Sullivan, W.C. (2002). “Views of Nature and Self-Discipline: Evidence from Inner City Children.” Journal of Environmental Psychology, 22, 49-63.

Added.

In response to comment by Yvain on Rational Home Buying
Comment author: gjm 26 August 2011 09:09:22PM *  10 points [-]

Since a house is inevitably both a place to live and an investment, it would seem to be appropriate to treat it as both. (Unless doing so spoils your enjoyment of it as a place to live, or something. For what it's worth, I've always thought of houses both ways, have never noticed such a negative effect, and have always been happy with the results on both counts. But I've been pretty fortunate.)

[EDITED to add: I agree that the extra mortgage interest you'll pay is a genuine extra cost -- and that, not the $35k price difference or whatever, is what you should be weighing against whatever you're paying the extra for. How the two figures relate to one another depends a lot on the mortgage interest rate, how quickly you repay, etc.]

In response to comment by gjm on Rational Home Buying
Comment author: Yvain 27 August 2011 09:17:16AM 0 points [-]

Good point, I'll edit.

Rational Home Buying

99 Post author: Yvain 27 August 2011 12:15AM

My parents are considering moving house. I've had a front-seat window to their decision process as they compare alternatives, and sometimes it isn't pretty.

A new house is one of the most important purchases most people will make. Because of the sums involved, the usual pitfalls of decision-making gain new importance, and it becomes especially important to make sure you're thinking rationally. Research in a couple of fields, most importantly positive psychology, offers some potentially helpful tips.

LOCATION, LOCATION, LOCATION

People so consistently under-count the pain of commuting when making choices that the problem has its own name: Commuter's Paradox. The paradox is that, although rational choice theory predicts people should balance commuting against other goods and costs, so that one person might have a longer commute but a nicer (or cheaper) house and so be just as happy overall, this doesn't happen: people who have long commutes are miserable, full stop. A separate survey by Kahneman and Krueger found that commuting was the least enjoyable of nineteen daily activities mentioned, and other studies have found relations between long commutes and poor social lives, poor health, high stress, and various other problems.

Psychologists aren't entirely sure why people so consistently under-count the pain of commuting. Maybe it's because it's viewed as "in-between" time rather than as an activity on its own; maybe it's because it comes in relatively short and individually bearable chunks repeated over many years, instead of as a single entity. In any case, unless you are mentally atypical you will probably have a tendency to undercount commute time when buying a new home, and may want to adjust for that tendency.

HOUSES COST A LOT OF MONEY

One of Kahneman and Tversky's famous bias experiments went like this: imagine you're buying a new shirt. It costs $40 at a nearby store, and it costs $20 at a store that's fifteen minutes away. Do you drive the fifteen minutes to save twenty bucks? Most people would.

Now imagine you're buying a new TV which costs $2020 at a nearby store, and $2000 at a store that's fifteen minutes away. Do you drive the fifteen minutes to save twenty bucks? Most people wouldn't.

In both cases, the tradeoff is the same - drive fifteen minutes to save twenty bucks - but people were much more willing to do it for the cheap item, because $20 was a higher percentage of its total cost. With the $2000 TV, the $20 vanishes into the total cost like a drop in the ocean and seems insignificant.

Nice homes can cost $500,000, $1,000,000, or even more. There doesn't seem to be a big difference in price between $710,000 and $745,000 houses; perhaps if the second home looked even a little nicer in an undefinable way you might be prepared to take it. But $35,000 is $35,000; if those minor advantages don't provide $35,000 worth of value, when measured on the same scale on which you measure the value of of movie tickets, shoes, and college funds, then you should buy the first house and keep the cash.

I find purchasing decisions easier when I think about them like this: which would you rather have, the second house, or the first house plus a two-week luxury vacation to anywhere in the world every summer for the next five years? The second house, or the first house plus a brand new Lexus? The second house and dining at home every week, or the first house and eating out at your favorite restaurant every weekend for the rest of your life? (EDIT: gjm points out that it's easier to resell houses than other types of good, so if you expect to resell your house you should really only be considering the extra money involved in the mortgage)

DON'T OVERCOUNT EASILY AVAILABLE DETAILS

The availability heuristic says that people overcount scenarios that are easy and vivid to imagine, and undercount scenarios that don't involve any readily available examples or mental images. For example, most people will assert, when asked, that there are more English words ending with "-ing" than with "-g". A moment's thought reveals this to be impossible - words ending in "-ing" are a subset of those ending in "-g" - but thinking specifically of "-ing" words makes it easier to bring examples to mind.

The real estate version of this fallacy involves exciting opportunities that you will rarely or never use. For example, a house with a pool may bring to mind the opportunity to hold pool parties. But most such plans will probably fall victim to akrasia, and even if they don't, how often can one person throw pool parties without exhausting their friends' interest? Pool parties may be fun to imagine, but they'll probably only affect a few hours every couple of months. Other factors, like the commuting distance and whether your children end up in a nice school, may affect several hours every day.

(a classic example here is the "extra bedroom for Grandma" - visits from Grandma are easy to imagine, but if she only comes a couple of days a year, spending tens of thousands more dollars for a house with an extra bedroom and bathroom for her is probably pretty stupid. You'd save money - and make her happier - by putting her up in the local five star hotel.)

LIGHT AND NATURE

Good illumination and a view of natural beauty aren't just pleasant luxuries, but can make important practical differences in your life.

Light, especially daylight, has a strong effect on mood. There are at least fifteen controlled studies showing that bright light reduces symptoms of seasonal and nonseasonal depression by about 10-20% over placebo. This is about equal benefit to some antidepressant drugs, and sufficient that light therapy is a recognized medical treatment for depression. Bright light leads to self-reported better mood even in subjects without a diagnosis of depression, and also leads to better sleep and more agreeable social interactions.

Light and nature have positive effects on health. Some of the most compelling data comes from hospitals, which have long realized that their patients near windows do better than their more interior counterparts. In one study, surgical patients near windows recovered faster (7.9 vs. 8.7 days), received fewer negative comments from nurses (1.1 vs. 4 notes), and needed fewer strong painkillers (1 vs. 2.5 doses) than matched controls without a view. Other studies have compared recovery of physiological indicators of stress (for example, blood pressure) in subjects viewing natural or artificial scenes; the subjects with views of nature consistently have healthier stress reactions.

Nature may have special benefits for children. Experiments with subjects of all ages and levels of mental health have shown nature increases mental functioning and concentration, but some of the most cited work has been in children with Attention Deficit Hyperactivity Disorder. Children who live in greener settings also (independently of wealth) do better on schoolwork and show greater ability to delay gratification. Large studies find with high certainty that students who take standardized tests in better lighting do up to 25% better than their literally dimmer schoolmates, and progress through lessons 15-25% faster.

You don't have to live in the Amazon to get a benefit: even children in a concrete building with a tiny "green island" boasting a single tree did better than their peers in a building without such an island.

BETTER FIRST IN A VILLAGE THAN SECOND IN ROME

Brains generally encode variables not as absolute values but as differences from an appropriate reference frame. That means that to really appreciate your wealth, you've got to be surrounded by people who are poorer than you are.

This seems to be empirically the case: a US study found the happiest Americans were rich people living in poor counties. However, this was true only of rich people living in rich neighborhoods of poor counties. As the study puts it, "individuals in fact are happier when they live among the poor, as long as the poor do not live too close".

Of course, this doesn't mean that you should move to Somalia for eternal bliss. There are community-wide benefits to living in a wealthy neighborhood, like better schools, and you may be better able to socialize with people from a similar class background as yourself. But given the choice between a neighborhood at the top of your price range and one at the bottom, you may find yourself more satisfied living in an area where it's the Joneses who have to try to keep up with you.

DON'T OVERSHOP AND DON'T OVERTHINK

It's easy to confuse "rationality" with a tendency to turn all decision-making over to conscious general-purpose reasoning, and in turn to assume that whoever ruminates about a decision the most is most rational. But there are at least two reasons to think that within reason it may be better to worry less over important decisions.

One is the finding that "comparison shopping" usually leads to less happiness in whatever you buy. Imagine being pretty sure you're going to buy House X until you look at House Y and find out that this one has a granite fireplace, and a pond in the backyard. It may be you don't like House Y at all - but now every time you go back to House X, you're thinking about how it doesn't have a granite fireplace or a pond, two features which you never would have even considered before. Whether you find this explanation plausible or not, the research generally agrees: too many choices result in less satisfaction with whatever you finally buy.

The second is the discovery that attempts to make your reasoning explicit and verbal usually result in worse choices. This includes that favorite of guidance counselors: to write out a list of the pros and cons of all your choices - but it covers any attempt to explain choices in words. In one study, subjects were asked to rate the taste of various jams; an experimental group was also asked to give reasons for their ratings. Ratings from the group that didn't need reasons correlated more closely with the ratings of professional jam experts (which is totally a thing) than those who gave justifications. A similar study found students choosing posters were more likely to still like the poster a month later if they weren't asked to justify their choice (Lehrer, How We Decide, p. 144).

The most plausible explanation is that having to verbalize your choices shifts your attention to features that are easy to explain in words (or perhaps which make good signaling value), and these are not necessarily the same features that are really important. In a telling experiment under the same protocol as the ones listed above, people asked to reflect upon their choices were more likely to choose the house with the extra room for Grandma than the house with the shorter commute times, because the extra reflection gave more opportunity for the availability heuristic to come into play.

CONCLUSION

Buying a house is one of the biggest decisions a family faces, and so has extra opportunity to be improved by rational thinking. Try to buy a house with good illumination and nearby green space in an area close to your workplace where you'll be relatively high on the social ladder. Carefully consider whether special features have genuine utility or are just highly available small details, and justify the relative differences in cost in absolute, not just relative terms. And, um, try to do all of this while following your gut instincts and not overshopping.

In response to Rational Home Buying
Comment author: gjm 26 August 2011 09:01:22PM 8 points [-]

The second house, or the first house plus a brand new Lexus?

Houses keep their value better than Lexuses. (Lexi. Lexoi. Lexūs. Whatever.) Eventually, you (or your heir(s)) will presumably sell the house, and then you will get more for the second house. Buying the first house gets you more liquidity than buying the second; it doesn't get you any more wealth. (If the housing market is performing well, which it did once upon a time, it may get you rather more in the long run.)

In response to comment by gjm on Rational Home Buying
Comment author: Yvain 26 August 2011 09:04:42PM 2 points [-]

Most of this assumes you're treating your house as a place to live and not as an investment. Because you'll probably be paying interest on a mortgage, it's usually not a smart idea to pay more than you need for a house - but I admit that the possibility of selling the house later is a major factor that alters a lot of these calculations.

Comment author: Vladimir_M 23 August 2011 11:10:51PM *  11 points [-]

Think of it as a conflict between a special moral module and general purpose reasoning. [...] The special moral module is what makes most people naturally deontologists instead.

I think that utilitarianism vs. deontology is a false dichotomy. People's natural folk ethics is by no means deontological -- refusing to break deontological rules in some situations where this is normally expected will also make you look weird, creepy, or even monstrous in the eyes of a typical person. As far as I see, virtue ethics is the only approach that captures the actual human moral thinking with any accuracy.

However, I agree with your remark if we replace deontology with virtue ethics. Where we might have a deeper disagreement is when the output of these special modules should be seen as baggage we'd better get rid of, and when it has non-obvious but vitally important functions.

You can end up utilitarian either because you're a psychopath and don't have the special moral module - in which case you default to general purpose reasoning - or because you're very philosophical and have a specific preference for determining moral questions by the same logic with which you determine everything else, thus deliberately overruling the special moral module.

My own hypothesis is that being very philosophical tends to produce primarily utilitarian signaling in the form of words and relatively cheap symbolic actions, and very little or no serious utilitarian behavior. And while some small number of people are persuaded by philosophical utilitarian arguments to undertake great self-sacrifice for (what they believe to be) the greater good, I doubt that anyone can be persuaded by such arguments to commit the utilitarian act in those "sacrificial" trolley-like scenarios. Therefore, if someone is observed to have acted in such a way, this would be strong evidence that it's due to antisocial traits, not philosophical inclinations.

Comment author: Yvain 24 August 2011 09:51:47AM 5 points [-]

I suppose I'd agree with you that folk ethics aren't exactly deontological, though I'd have trouble calling them virtue ethics since I don't understand virtue ethics well enough to draw any predictive power out of it (and I'm not sure it's supposed to have predictive power in moral dilemmas). Maybe you're right about the distinction between folk moral actions and folk moral justifications - in the latter, people seem much more supportive of deontological justifications than utilitarian justifications, but I don't know how much effect that has on actual actions.

My own hypothesis is that being very philosophical tends to produce primarily utilitarian signaling in the form of words and relatively cheap symbolic actions, and very little or no serious utilitarian behavior.

Do you think this is specific to utilitarianism or more of a general issue with philosophy? David Hume didn't seriously stock up on candles in case the sun didn't rise the next morning, Objectivists probably do as many nice things for other people as anyone else, and economists don't convert en masse even though most don't have a good argument against stronger forms of Pascal's Wager. I don't really expect thoughts to influence ingrained behaviors that much, so it doesn't seem to require any special properties of utilitarianism to explain this.

Where we might have a deeper disagreement is when the output of these special modules should be seen as baggage we'd better get rid of, and when it has non-obvious but vitally important functions.

I'm not sure to what degree we disagree on that.

I would agree that the special modules have "important functions", but I would cash out "important" in a utiltiarian way: it would require an argument like "If we didn't have those modules people would do crazy things and society would collapse, which would be bad". This seems representative of a more general sense in which to resolve conflicts in our special moral reasoning we've got to apply general reasoning to them and utilitarianism is sort of the "common currency" that allows us to do that. Once we've done that we can link special moral reasoning to our more general reasoning and ground a lot of our intuitive moral rules. This is in the same sense that our visual processing modules are a heck of a lot better than trying to sort out luminance data from the environment by hand, but we still sometimes subject it to general-purpose reasoning when eg we're not sure if something is an optical illusion.

Comment author: Yvain 23 August 2011 09:53:22PM 15 points [-]

Luke quoted Joshua Greene as saying:

...deontological judgments tend to be driven by emotional responses, and... deontological philosophy, rather than being grounded in moral reasoning, is to a large extent an exercise in moral rationalization. This is in contrast to consequentialism, which, I will argue, arises from rather different psychological processes, ones that are more 'cognitive,' and more likely to involve genuine moral reasoning...

If this is true, then it makes sense that people who don't have emotional responses to moral questions won't be deontologists.

Think of it as a conflict between a special moral module and general purpose reasoning. General purpose reasoning that you'd use in eg economics tells you that if you lose $100 to gain $200, you come out ahead - it's utilitarian. The special moral module is what makes most people naturally deontologists instead.

You can end up utilitarian either because you're a psychopath and don't have the special moral module - in which case you default to general purpose reasoning - or because you're very philosophical and have a specific preference for determining moral questions by the same logic with which you determine everything else, thus deliberately overruling the special moral module.

Comment author: Tesseract 27 July 2011 03:39:40PM 0 points [-]

FYI, I showed the manual to a (non-Less Wrong) philosophy-major friend who runs D&D games, so you may develop a splinter group.

Comment author: Yvain 20 August 2011 10:20:28PM 0 points [-]

If he wants to conduct a game, have him contact me so I can send him the DM Guide.

Comment author: Vladimir_Golovin 20 August 2011 07:03:53AM 0 points [-]

Is there a connection between the time discounting curve and hand grip duration?

Comment author: Yvain 20 August 2011 10:02:42AM 0 points [-]

Trading off pain/effort while gripping for...well, whatever it was that made people want to grip the handgrip in the first place. The feeling of having been helpful in the study? Personal pride at believing yourself to have high willpower?

Comment author: Yvain 19 August 2011 04:29:43PM 2 points [-]

I'm pretty convinced that this is just a result of glucose shifting the time discounting curve. No "expendable resource" required.

I know there's at least one study that purports to show otherwise, but this explanation is so much more elegant than any other I've yet seen proposed that right now I'm defying the data until this whole area is better understood.

Comment author: cousin_it 12 August 2011 07:50:40AM *  1 point [-]

Hm, I wasn't joking actually.

Maybe "self-control" was a poorly chosen name for that worksheet item. How about we call it "the skill of working without interruptions to tick some boxes off a worksheet?" That seems relevant to many people, and skipping it doesn't sound like a reasonable cost-benefit decision :-)

Comment author: Yvain 12 August 2011 03:37:55PM 0 points [-]

See my reply here

Comment author: Pfft 12 August 2011 01:31:43AM *  2 points [-]

I don't understand why you are singling out the self-control checkbox for this criticism, when it seems to apply equally well to all the other checkboxes also. The closest other item seems to be the endurance one: for me keeping running and keeping working feels subjectively very similar. Both or them would be easier to do if someone pointed a gun at you, both of them depend on how much you care about the worksheet (and also about more inherent rewards, like getting fit, experiencing "runner's high", getting work done, experiencing flow state), and both of them will be easier or more difficult for different people.

Your second point, than some people find it easier to concentrate than others, seems to be an argument in favour of considering it a real variable across people?

Maybe what you are objecting against is the name of the item, "self control", since you think it is not philosophically sound? But things should add up to normality. Being about to concentrate on work seems like an obviously desirable ability. If someone (like me, currently) can't do it, then it seems very reasonable to want to learn how. Judging by the number of akrasia-posts on Less Wrong, lots of other people think so too. If we currently have no good philosophical analysis of this, then by all means let us try to analyze it better, but let's not deny that the phenomenon exists.

Comment author: Yvain 12 August 2011 03:35:55PM *  9 points [-]

You're right, I did a terrible job of trying to verbalize the reasons behind my intuition that the self-control requirement is too vague. Let me try again, starting by talking about the difference between requirements where you have to estimate something, versus requirements where you have to perform something.

For example, I have no problem with the "run a mile" requirement. I don't care enough about "leveling" to go out and run a mile right now just to prove I can, but I don't have to - I ran several miles yesterday for non-leveling related reasons, so I know I could do it if I wanted to. The same is true of most of the others: from past experience, I already have probabilities > 80% I could do memory, finance, and creativity; I have a similarly high probability that I couldn't bake pancakes without gaining new knowledge, and I'm not sure about the strength and programming ones but it would be very easy to find out.

The self-control requirement is different. Could I do it if my life were at stake? I'm near-100% sure I could. Would I do it for the sake of this leveling game? Empirically, no. But that only puts it in the same category as running, which I also wouldn't do for the sake of this leveling game but which I know I have the necessary skill in. And it may be that the world's greatest expert in self-control, let's say the Buddha, also would not do that requirement for the sake of this leveling game (the Buddha cares not for human status and awards).

So we either have to bite the bullet and say that means the Buddha has low self-control, or we have to say "Yeah, but if the Buddha had sufficient incentive, he would do it, so he qualifies". But in that case, alcoholics also qualify, since if they had sufficient incentive, they would presumably quit drinking.

(for a metaphor, consider a Finances category in which the requirement is "Burn $1000 in bills". This test is statistically specific for well-off people - only those with at least a spare $1000 could do it, and the more money you have the easier it is to complete - but it's not statistically sensitive - many people who have the extra money will nevertheless choose not to do it.)

This seems to get a bit closer to why this item confuses me.

Comment author: cousin_it 10 August 2011 07:10:51AM 12 points [-]

It also threatens to conflate how skilled you are at things with how much you're willing to do something just to tick off a box on a worksheet.

For the skill of self control, is there a difference between these two things? :-)

Comment author: Yvain 11 August 2011 04:44:28PM 5 points [-]

I know you're probably joking, but I think this is an important point, so I want to say - yes!

If I ask you to go without food for seven days, without really giving you a reason, but just saying it would increase my respect for you or something -- and you refused, would that be the fault of your poor self-control? No, it would be a reasonable cost-benefit decision.

On the other hand, when we take something like addiction that people commonly claim to have no self-control over - well, if you put a gun to someone's head and threatened to kill them if they had another drink of alcohol, then as long as you can continue enforcing the threat, suddenly self-control isn't so much of an issue.

Now all of this is complicated by hyperbolic discounting and psychodynamics and so on, but at some point, "self control" is a matter of how much of a reward or punishment you're expecting. So if you keep that on there, I predict you're measuring two things. First of all, how much people care about a worksheet - such that if this "leveling" thing became wildly popular and prospective employers asked you your level before hiring that would change motivation. And how annoying it is for you to concentrate at that level (eg ADHD people would have a much harder time; other people might have more or less natural concenration issues) - such that if you chose a different "self control" task like going without food for a certain amount of time, or squeezing a lever at a certain strength, you would get different results.

In either case, I don't think measuring "self-control" as a real variable across people is on a firm philosophical footing.

Comment author: Yvain 10 August 2011 04:22:07AM *  16 points [-]

The biggest problem here seems to be that a lot of these things would be pretty annoying even if you had well beyond the skills necessary to accomplish them.

Self-control and Social seem like the worst offenders here, especially insofar as it's harder to dismiss them as something you already know you can do (I already know I can run a mile or what passages I have memorized, but I don't count how many times per month I meet with friends, or how many times per month I work two hours straight).

It also threatens to conflate how skilled you are at things with how much you're willing to do something just to tick off a box on a worksheet. Imagine someone arranging four meetups, then calling back and saying "Actually, I'm not interested in meeting up at all, I just asked you to meet up to prove I could do it, for this online worksheet thing. Anyway, I'll talk to you next time I have some real plans, and sorry about the inconvenience."

(if you don't believe anyone could possibly be that pathetic, consider that it was the first thought that came into my mind when I saw that requirement.)

This is even more true of self-control. Anyone could work two hours straight to save their own life, most people could work two hours straight if there's a big project due the next day, but I don't know how many people could work two hours straight because a leveling worksheet told them to.

Comment author: Yvain 08 August 2011 07:48:11AM 20 points [-]

I don't know your mind, but I would predict that in the average person, determination to donate ~100% of earnings to a good cause will not persist for 18 years. If you think there's a pretty good chance that you'll end up changing your mind about that, might as well give the money now while you're still a perfect utilitarian (for which I admire you a huge amount, by the way) rather than give nothing for years and years and end up not giving anything at all.

I'm currently in my last year of medical school pursuing a career in psychiatry. If you have any questions on the process, feel free to ask me by PM or email (scott@shireroth.org).

Comment author: Wei_Dai 24 July 2011 01:04:43PM 3 points [-]

I'm still confused about what point Yvain might be making by substituting "tendency" for "intuition" in this formulation of reflective equilibrium. I can think of two possibilities, but neither of them seems like something he might endorse.

  1. When we reflect on what we really want, we should take into consideration not just our intuitions, but our behavioral tendencies. (But Yvain previously wrote "NO NEGOTIATION WITH UNCONSCIOUS".)
  2. After we've reached reflective equilibrium, our behavioral tendencies can be said to be our preferences. (But suppose I have a tendency to reflexively slap myself whenever I see the color red. After reflecting on it, I decided this is not something I should do, but I have no power to actually edit the reflex away. Based on what Yvain wrote in The Blue-Minimizing Robot, I don't think he would say that I actually do prefer to slap myself (or have myself slapped) whenever the color red is in my field of vision.)

The only other explanation I have is that Yvain has gotten used to writing "tendency" for his other recent posts, and kept using it even when it no longer makes sense to. Does anyone else have other ideas?

Comment author: Yvain 01 August 2011 06:28:04PM 0 points [-]

I didn't have any particular interesting agenda for that word choice. If I had to justify it, I would say that to me "intuition" implies a belief (for example, I have an intuition that people who steal more money ought to be punished) and "tendency" implies an action (for example, when asked how much to punish a thief, I might respond "five years"). I am trying to carefully avoid language that implies the existence of beliefs, not because I have strong opinions on the matter but because I'm unsure.

Comment author: Yvain 31 July 2011 08:30:31AM *  39 points [-]

The truth is that people aren't anything like coherent enough to refuse cryonics for a reason like that.

I agree with almost all of what you say about no grand narrative and mostly just conformity, but I'm not willing to entirely dismiss this explanation as even a small part of the puzzle. It doesn't seem much different than the theories that poor people with few life prospects have higher temporal discount rates and are more likely to engage in risky/criminal behavior because they have less to protect. People aren't coherent enough to think "Well, stealing this watch has a small probability of landing me in prison, but my life now isn't so satisfying, so I suppose it's worth the risk, and I suppose it's worth risking a lot later for a small gain now since I currently have so little", but there's some inner process that gives more or less that result.

If even the few people who get past the weirdness factor flinch away from the thought of actually being alive more, I expect that would make a significant difference.

I'm going to try a test question that might differentiate between "cryonics sounds weird" and "I don't like life enough to want to live even more" on my blog. Obviously no one from here post on that since you already know where it's going.

If you want people to sign up for cryonics, the method with by far the strongest conversion ratio is to train them from scratch in advanced sanity techniques.

Alternate hypotheses: your followers are mostly technophile singularitarians, and technophile singularitarians are attracted to cryonics independently of rationalist training. Your followers believe there may be a positive singularity, which means the future has a reason to be much better than the present and avoid the unpleasantness Darwin describes in the article. Your followers are part of maybe the one community on earth, outside the cryonics community itself, where the highest-status figures are signed up for cryonics and people are often asked to justify why they have not done so. Your followers are part of a community where signing up for cryonics signals community affiliation. Your followers have actually heard the arguments in favor of cryonics and seen intelligent people take them seriously, which is more than 99.9% of people can say.

Comment author: Yvain 01 August 2011 06:37:20AM 18 points [-]

Judging by the experiment with the secretly identical question, I seem to have been wrong. Everyone says they would jump at the chance to be reincarnated, so lack of desire to live longer apparently doesn't play as significant a role in cryonics refusal as I thought.

Comment author: Vladimir_M 29 July 2011 10:09:35PM *  12 points [-]

I agree that even a post-scarcity society would need some form of employment to determine status and so on. But that seems irrelevant to the current problem: one where even people who are not interested in status need to work long hours in unpleasant conditions just to pay for food, housing, and medical costs, and where ease of access to these goods hasn't kept pace with technological advantages.

But that's not the case in the modern developed world. If you are really indifferent to status, you can easily get enough food, housing, and medical care to survive by sheer freeloading. This is true even in the U.S., let alone in more extensive welfare states.

Of course, completely forsaking status would mean all sorts of unpleasantness for a typical person, but this is only because we hate to admit how much our lives revolve around zero-sum status competitions after all.

I think a genuinely post-work society would have its own ways of producing status based on hobbyist communities, social interaction, and excellence at arts/scholarship/sports/hobbies; the old European nobility was able to handle its internal status disputes in this way, though I don't know how much fo that depended on them knowing in the back of their mind they were all superior to the peasantry anyway.

Don't forget about the status obtained from having power over others. That's one part of the human nature that's always dangerous to ignore. (The old European nobility was certainly not indifferent to it, and not just towards the peasants.)

Also, there would always be losers in these post-work status games who could improve their status by engaging in some sort of paid work and saving up to trade for the coveted status markers. These tendencies would have to be forcibly suppressed to prevent a market economy with paid labor from reemerging. It's roughly analogous to the present sexual customs and prostitution. Men are supposed to find sexual partners by excelling in various informal, non-monetary status-bearing personal attributes, but things being zero-sum, many losers in this game find it an attractive option to earn money and pay for sex instead, whether through out-and-out prostitution or various less explicit arrangements.

Comment author: Yvain 31 July 2011 08:44:27AM 17 points [-]

But that's not the case in the modern developed world. If you are really indifferent to status, you can easily get enough food, housing, and medical care to survive by sheer freeloading. This is true even in the U.S., let alone in more extensive welfare states.

I'm not sure this is true; I know little about welfare politics, but I was under the impression there was a major shift over the last ten years toward limiting the amount of welfare benefits available to people who are "abusing the system" by not looking for work.

One could probably remain alive for long periods just by begging and being homeless, but this raises the question of what, exactly, is a "life worth living", such that we could rest content that people were working because they enjoy status competitions and not because they can't get a life worth living without doing so.

This is probably way too subjective to have an answer, but one thing that "sounds right" to me is that the state of nature provides a baseline. Back during hunter-gatherer times we had food, companionship, freedom, et cetera without working too hard for them (the average hunter-gatherer only hunted-gathered a few hours a day). Civilization made that kind of lifestyle impossible by killing all the megafauna and paving over their old habitat, but my totally subjective meaningless too-late-at-night-to-think-straight opinion is that we can't say that people can opt-out of society and still have a "life worth living" unless they have it about as good as the hunter-gatherers they would be if society hadn't come around and taken away that option.

The average unemployed person in a developed country has a lot of things better than hunter-gatherers, but just the psychological factors are so much worse that it's no contest.

Comment author: Eliezer_Yudkowsky 30 July 2011 06:13:05PM 46 points [-]

This is a fantastically burdensome explanation for why people don't sign up for cryonics. Do people who do sign up for cryonics usually have happier lives? (Not that I've heard of.) Do the same people who turn down cryonics turn down other forms of medical care? (Not that I've heard of.) If we found that people signing up for cryonics were less happy on average, would we be able to construct an equally plausible-sounding symmetrical argument that people with happy, fulfilled lives see no need for a second one? (Yes.)

I hate to go into psychologizing, but I suspect that Mike Darwin wants a grand narrative of Why, Oh Why Cryonics Fails, a grand narrative that makes sense of this shocking and incomprehensible fact and gives some info on what needs to be done to relieve the frustration.

The truth is that people aren't anything like coherent enough to refuse cryonics for a reason like that.

Asking them about cryonics gets their prerecorded verbal behaviors about "immortality" which bear no relation whatsoever to their feelings about whether or not life is fun.

Remember the fraction of people that take $500 for certain over a 15% chance of $1 million? How could you possibly need any elaborate explanation of why they don't sign up for cryonics? Risk-aversion, loss-aversion, ambiguity-aversion, status quo bias.

Cryonics sounds strange and not-of-our-tribe and they don't see other people doing it, a feeling expressed in words as "weird". It's perceptually categorized as similar to religions or other scams they've heard about from the newspaper, based purely on surface features and without any reference to, or remediability by, the strength of the underlying logic; that's never checked. Mike Darwin thinks that if you have better preservation techniques, people will sign up in droves, because right now they're hearing about cryonics and rejecting it because the preservation techniques aren't good enough. This is obviously merely false, and the sort of thing which makes me think that Mike Darwin needs a grand narrative which tells him what to do to solve the problem, the way that Aubrey de Grey thinks that good enough rejuvenation results in mice will grandly solve deathism.

I recently got a phone call saying that, if I recall correctly, around a quarter - or maybe it was half - of all Alcor's cryonics signups this year, are originating from LW/Yudkowsky/rationality readers. If you want people to sign up for cryonics, the method with by far the strongest conversion ratio is to train them from scratch in advanced sanity techniques. Nothing else that cryonics advocates have tried, including TV ads, has ever actually worked. There's no simple reason people don't sign up, no grand narrative, nothing that makes sense of cryonicists' frustration, people are just crazy in rather simple and standard ways. The only grand narrative for beating that is "soon, your annual signups will equal 10% of the people who've gone through a rationality bootcamp plus 1% of the people who've read both Eliezer's nonfiction book and Harry Potter and the Methods of Rationality."

Comment author: Yvain 31 July 2011 08:30:31AM *  39 points [-]

The truth is that people aren't anything like coherent enough to refuse cryonics for a reason like that.

I agree with almost all of what you say about no grand narrative and mostly just conformity, but I'm not willing to entirely dismiss this explanation as even a small part of the puzzle. It doesn't seem much different than the theories that poor people with few life prospects have higher temporal discount rates and are more likely to engage in risky/criminal behavior because they have less to protect. People aren't coherent enough to think "Well, stealing this watch has a small probability of landing me in prison, but my life now isn't so satisfying, so I suppose it's worth the risk, and I suppose it's worth risking a lot later for a small gain now since I currently have so little", but there's some inner process that gives more or less that result.

If even the few people who get past the weirdness factor flinch away from the thought of actually being alive more, I expect that would make a significant difference.

I'm going to try a test question that might differentiate between "cryonics sounds weird" and "I don't like life enough to want to live even more" on my blog. Obviously no one from here post on that since you already know where it's going.

If you want people to sign up for cryonics, the method with by far the strongest conversion ratio is to train them from scratch in advanced sanity techniques.

Alternate hypotheses: your followers are mostly technophile singularitarians, and technophile singularitarians are attracted to cryonics independently of rationalist training. Your followers believe there may be a positive singularity, which means the future has a reason to be much better than the present and avoid the unpleasantness Darwin describes in the article. Your followers are part of maybe the one community on earth, outside the cryonics community itself, where the highest-status figures are signed up for cryonics and people are often asked to justify why they have not done so. Your followers are part of a community where signing up for cryonics signals community affiliation. Your followers have actually heard the arguments in favor of cryonics and seen intelligent people take them seriously, which is more than 99.9% of people can say.

Comment author: Carinthium 27 July 2011 06:58:10AM 1 point [-]

So, when does this start?

Comment author: Yvain 30 July 2011 06:18:04AM 0 points [-]

Still waiting on a few people's email addresses.

Comment author: novalis 28 July 2011 06:25:31AM 1 point [-]

Can the automatic expiration of Enantiomerize kill a creature? For instance, if a creature is in front of me, and I enantiomerize it to behind me, and then I take a step forward, causing in front of me to mean off a cliff...

Comment author: Yvain 30 July 2011 04:40:21AM 0 points [-]

I expect the ending of the spell would bring it back to its original position, but I guess the text doesn't specify that.

Comment author: Vladimir_M 29 July 2011 03:16:36AM *  34 points [-]

I was reading a prediction from fifty years ago or so that by 2000, people would only work a few hours a day or a few days a week, because most work would be computerized/roboticized and technology would create amazing wealth. Most work has been computerized/roboticized, technology has created amazing wealth, but working conditions are little better, and maybe worse, than they were fifty years ago.

Technological advances can't shorten the work hours because even in a society wealthy and technologically advanced enough that basic subsistence is available for free, people still struggle for zero-sum things, most notably land and status. Once a society is wealthy enough that basic subsistence is a non-issue, people probably won't work as much as they would in a Malthusian trap where constant toil is required just to avoid starvation, but they will still work a lot because they're locked in these zero-sum competitions.

What additionally complicates things is that habitable land is close to a zero-sum resource for all practical purposes, since to be useful, it must be near other people. Thus, however wealthy a society gets, for a typical person it always requires a whole lot of work to be able to afford decent lodging, and even though starvation is no longer a realistic danger for those less prudent and industrious in developed countries, homelessness remains so.

There is also the problem of the locked signaling equilibrium. Your work habits have a very strong signaling component, and refusing to work the usual expected hours strongly signals laziness, weirdness, and issues with authority, making you seem completely useless, or worse.

As for working conditions, in terms of safety, cleanliness, physical hardship, etc., typical working conditions in developed countries are clearly much better than fifty years ago. What arguably makes work nowadays worse is the present distribution of status and the increasing severity of the class system, which is a very complex issue tied to all sorts of social change that have occurred in the meantime. But this topic is probably too ideologically sensitive on multiple counts to discuss productively on a forum like LW.

Comment author: Yvain 29 July 2011 09:04:06PM *  26 points [-]

I agree that even a post-scarcity society would need some form of employment to determine status and so on. But that seems irrelevant to the current problem: one where even people who are not interested in status need to work long hours in unpleasant conditions just to pay for food, housing, and medical costs, and where ease of access to these goods hasn't kept pace with technological advantages.

And although I don't think it quite related, I am less pessimistic than you abou the ability of a post-scarcity society to deal with land and status issues. Land is less zero-sum than the finitude of the earth would suggest because most people are looking not for literal tracts of land but for a house in which to live, preferably spacious - building upward, or downward as the case may be, can alleviate this pressure. I'm also not convinced that being near other people is as big a problem as you make it out to be: a wealthier society would have better transportation, and cities have enough space to expand outward (giving people access to other humans on at least one side) almost indefinitely. There will always be arbitrarily determined "best" neighborhoods that people can compete to get into, but again, this is a totally different beast from people having to struggle to have any home at all.

I think a genuinely post-work society would have its own ways of producing status based on hobbyist communities, social interaction, and excellence at arts/scholarship/sports/hobbies; the old European nobility was able to handle its internal status disputes in this way, though I don't know how much fo that depended on them knowing in the back of their mind they were all superior to the peasantry anyway.

Agreed that the class system is an important and relevant issue here.

Comment author: multifoliaterose 29 July 2011 02:25:58AM *  11 points [-]

Work is terrible, and the lives of many working people, even people with "decent" jobs in developed countries, are barely tolerable. It is currently socially unacceptable to mention this. Anyone who breaks that silence has done a good deed.

How confident are you that this reflects the experience of working people rather than how you would feel if you were in their position?

I've wondered about this a lot myself. Note along with figure 3 of the quoted article, according to a Gallup poll the average self reported life satisfaction in America is around 7/10. Presumably this average includes even including the sick/elderly/poor. I believe that my own self reported life satisfaction would be considerably lower than that if I were living the life of an average American.

I would guess that the difference is mostly accounted for by my own affective response to a given situation diverging heavily from the affective response that members of the general population would have in the same situation.

Comment author: Yvain 29 July 2011 08:48:02PM 18 points [-]

How confident are you that this reflects the experience of working people rather than how you would feel if you were in their position?

Somewhat confident. I work at a medical clinic. The number of people who come in with physical complaints relating to their job, psychological/stress complaints relating to their job, or complaints completely unrelated to their job but they talk to the doctor about how much they hate their job anyway because he's the only person who will listen - is pretty impressive.

Comment author: Yvain 29 July 2011 12:30:20AM 65 points [-]

Upvoted for several reasons:

  • excellent theory about cryonics, much more plausible than things like "people hate cryonics because they're biased against cold" that have previously appeared on here.

  • willingness to acknowledge serious issue. Work is terrible, and the lives of many working people, even people with "decent" jobs in developed countries, are barely tolerable. It is currently socially unacceptable to mention this. Anyone who breaks that silence has done a good deed.

  • spark discussion on whether this will continue into the future. I was reading a prediction from fifty years ago or so that by 2000, people would only work a few hours a day or a few days a week, because most work would be computerized/roboticized and technology would create amazing wealth. Most work has been computerized/roboticized, technology has created amazing wealth, but working conditions are little better, and maybe worse, than they were fifty years ago. A Hansonian-style far future could lead to more of the same, and Hanson even defends this to a degree. In my mind, this is something futurologists should worry about.

  • summary of the article was much better than the article itself, which was cluttered with lots of quotes and pictures and lengthiness. Summaries that are better than the original articles are hard to do, hence, upvote.

Comment author: Yvain 28 July 2011 04:26:44AM 0 points [-]

Karl, Manfred, muflax, zefreak, Carinthium, Emile - and anyone else who's interested in participating but hasn't said so yet - please send me your email, either by posting it here, by PMing it to me, or by sending it to scott[at]shireroth[dot]org.

Comment author: Bongo 24 July 2011 04:14:10PM *  10 points [-]

I don't know how much sense the real-world tropes of skeptical atheists and fervently faithful theists make in a world where you can literally bargain with God to get your dead friend back from Heaven. In the D&Dis world, it really is atheism that requires faith!

Comment author: Yvain 25 July 2011 04:41:09AM 2 points [-]

In the campaign, the atheists are trying to fight/destroy the gods, who they believe are false gods distracting from the worship of the true gods Truth and Wisdom. I didn't want to make that too obvious in the book because it might limit the usefulness of the classes in other settings.

Comment author: [deleted] 24 July 2011 01:45:13PM *  15 points [-]

(Gah, denial of service attack detected! Roll for willpower or lose two hours of your life! ... Failed, dammit.)

Very nice. I remember an older attempt at a D&Dis system soon after the comic came out, but yours is much more unified and internally consistent. And all classes have lots of nice spells and the pun level is adequate. Looks fun.

Before reading, I thought, "Hey, let's try to roll a Discordian!", but then I wondered what I have to choose to cast "dispel colonialism". I guess I'll have to enter the dreaded halls of the continentals.

(This is a bit of a pet peeve. Don't take this as serious criticism. It's not even representative of the whole work, but if I go continental, I might as well quote-mine, summon the subtext and cast Wall of Text to further my own agenda.)

Logic is the purest of the disciplines, and does not permit alternate opinions; all students of logic learn the same spells regardless of their beliefs.

Dialetheists would like to have a word with you. (And its not just an obscure position. Many Mahayana philosophers have fully embraced it for centuries.)

Asians may follow non-dualist philosophies like Buddhism or Taoism that eschew the material world.

First of, Asian? The bloody European traditions get 3 out of 5 races, but Asian philosophy gets one?! The Confucians, Vedantists, Jains and whatnot really appreciate being lumped with the depressing Buddhists and crazy Taoists. Ancient Indian philosophy alone already had all the other classes (well, maybe not Objectivism). If you mean non-dual, just say that.

(And Hades, throwing the Greeks together also just... makes my eyes twitch. I'm really not a fan of lumping philosophies by whatever geography made it most prominent (in the West). "Anglo-American" and "Continental" is fine, as both have a fairly unified attitude and there are no better labels. But Greeks and Asians are way too diverse, especially when you identify them with one example that has nothing in common with other, equally important traditions. "Aristotelean" and "Non-dual" or something like that would be better.)

And Taoism is not really non-dual. I can see why you would mention it, as some branches are (particular the visible western ones), but it has the frigging yin-yang as its symbol for a reason. Also, Taoists have spent most of history trying to become physically immortal and to have as much fun in the process as possible. They are outright (physical) hedonists. There is nothing about renunciation in Taoism at all.

Greeks are the original philosophers.

As someone learning Akkadian, I'll put on my philosophy hipster hat and would like to remind you that, as a rule of thumb, everything is Older Than You Think. "Archetypal" philosophers, alright. They really made the profession cool. But "original"?

(Loving the Elves, though.)

If Theist: Catholicism | Protestantism | Syncretism |Fideism | Evidentialism | Serve God | Serve Mammon

I understand that this list is not meant to be complete at all, but theism = Christianity? At least a token polytheist or something? I'm way too underground for this Jesus stuff, I totally want a Heavenly Court or some dragons. You cannot be good without dragons.

In D&Dis, Absolutists get -2 morality, and Anarchists get -1 phronesis.

It's nice to see that in Sophia, the Anarchist scare is over.

DERRIDA'S DECONSTRUCTION HEX (4B)
Cast during an argument between a player and the GM over a rule interpretation. You analyze the rulebook for hidden meanings that may be opposite to the apparent intent of the text, ensuring victory to your favored interpretation of the rules. GM can nullify this spell at his discretion if abused. Starting an argument over otherwise clear rules just so you can use this spell is definitely an abuse.

So you favor a totalitarian GM, a gatekeeper of the Truth? Any criticism can be dispelled as attacking a "clear" rule (ignoring that the existence of the attack and the viability of an alternative interpretation question this "clarity")? What, exactly, did this GM do to gain privileged access to Truth? And what constitutes "abuse" depends on the abuser and the abused and their socio-cultural frameworks. Simply silencing an interpretation as "abuse" is an exertion of power, not an argument. Any power must be questioned. You are clearly including this exception so you can protect your pet weltanschauung and existing power structure. Your subtext speaks louder than your text. Also before, Nietzscheans are the only ones to summon dragons, without which you can't be good, so you give away your immorality. That you give Absolutists a -2 morality is clearly just signaling. You imperialist. (Am I doing this continental thing right? Do I need more being?)

BANISH P-ZOMBIE (R)

But... how do I know who I can cast this on?

Looking through your item list, I'm kinda divided. On the one hand, it makes sense, is funny and represents all the branches I wish would constitute philosophy. On the other hand, I see only a single continental book. There really should be a Heidegger or Hegel. How else am I to cast confusion, negate any spells into their synthesis or get into a greater state of dasein?

Materials are not priced like other items: the exact price of an object depends on the size of the object, the quality of the material, and the purpose to which it is being used. The price mentioned here is only a guide for the DM to determine context-appropriate prices for these objects - if you can find them at all!

You mean the price depends on the amount of labor that went into their production, right? </marxist>

In response to comment by [deleted] on Dungeons and Discourse implementation
Comment author: Yvain 25 July 2011 04:38:59AM *  3 points [-]

Most of the races were taken straight from the comics, but I endorse them. It's like how D&D has a race "elf", even though experts can distinguish betweeen the Calaquendi and the Moriquendi and so on.

I agree a lot of the options are very limited (you didn't even protest the division of morality into Kantian vs. Utilitarian! That's the worst!) because I wanted to create separate game mechanics and possibilities for each choice. If there were many different religions included, I don't know how the Apologist class would look, but it would have to be pretty different from any of the other classes, and much more complex. And I wanted to make the system resemble the debates that most potential players would be a part of, and those are more likely to be Christian vs. atheist with a little bit of Zen Buddhism or something on the side than Mahayana vs. Theravada, or Gelug vs. Nyingma. And if I were to put in Gelug vs. Nyingma, I'd have to put it into the campaign somehow, and that would confuse 99% of people. So yeah, it's very much western philosophy, and simplified Western philosophy at that.

...and as for the Continentals, I just don't know enough about them to give them justice, even enough justice to mock them. I was going to have Heidegger's Being and Time as a book with the power to Confuse every character within a certain radius, but then I realized I would give anything by Hegel exactly the same power, and so on.

...maybe you should write the first expansion book.

I'll see if I can respond to some of your other issues at more length later.

Comment author: Bongo 24 July 2011 01:19:48PM *  5 points [-]

Looks awesome. Some errata:

  • bottom of page 7 says Cartesian doubt is 3 speed and 1 rationality, while the list on page 13 says it's 3 speed and 0 rationality.
  • second paragraph on page 7 says "cast two squares and then cast the spell".
  • page 59 lists LHP things for RHP, where it says "giving you"
  • page 89 says "PROBABILITY THEORY: THE LANGUAGE OF SCIENCE" whereas it's actually the logic of of science.
Comment author: Yvain 25 July 2011 04:34:09AM 2 points [-]

Good catches. Thanks,

Comment author: gwern 24 July 2011 08:01:06PM 1 point [-]

Perhaps I am missing the joke, but the Seed AI seems to be missing from the Summons section.

Comment author: Yvain 25 July 2011 04:33:50AM 2 points [-]

Was removed to make it less predictable in campaign.

Dungeons and Discourse implementation

24 Yvain 24 July 2011 10:37AM

I've been working on an unauthorized implementation of Dresden Codak's Dungeons and Discourse, a fictional role-playing game that combines philosophy and high fantasy. You can find a very error-ridden, but possibly usable, rough draft of it at http://www.raikoth.net/Stuff/ddisplayer.pdf. Yes, obviously this is crazy and I have no life. There is no need to point that out further.

I'd like to try to run a campaign. It would be maybe an hour or two a week on IRC, and subject to my schedule, which is terrible and can include disappearing for months at a time (in particular I probably won't have internet access in August). Still, I would like to at least gauge interest and start some preliminaries now. And if anyone wants to run a campaign IRL at a meetup group or something, I can send them the file with the campaign walkthrough, though I'm not sure how much I would recommend it at this point.

Anyone who's interested in participating please let me know (especially if you have philosophical beliefs wildly different from the standard Less Wrong hive mind, or if you know any interested parties who do, since the game would be dreadfully boring if everyone agreed on everything or for that matter anything). Also, I suppose if people want to record the errors and contradictions and non sequiturs and exploits in the manual you might as well post them here so I can fix them.

Comment author: Yvain 23 July 2011 10:25:13AM 10 points [-]

Why do we see this 'Time-Ask Effect'? Perhaps it is because thinking about spending time on something activates a mindset of emotional meaning and satisfaction, allowing a donor to connect emotionally with a charity, whereas thinking about spending money activates a purely instrumental mindset.15 Whatever the reason, asking for time before money may result in more of both.

I must be more cynical than you. I'd think that if people said "yes", then they've already committed themselves to the organization and so would to give money, and/or if they said no, they would be feeling unpleasantly non-altruistic and would give money to assuage their conscience. Did the studies show differences in the money-ask broken down by whether they said yes or no to the time-ask?

Also, is there any data on whether people feel happier only after donating to fuzzy charities like the local animal shelter, or whether they'll also feel happier donating to something very abstract like SIAI?

Tendencies in reflective equilibrium

27 Yvain 20 July 2011 10:38AM

Consider a case, not too different from what has been shown to happen in reality, where we ask Bob what sounds like a fair punishment for a homeless man who steals $1,000, and he answers ten years. Suppose we wait until Bob has forgotten that we ever asked the first question, and then ask him what sounds like a fair punishment for a hedge fund manager who steals $1,000,000, and he says five years. Maybe we even wait until he forgets the whole affair, and then ask him the same questions again with the same answers, confirming that these are stable preferences.

If we now confront Bob with both numbers together, informing him that he supported a ten year sentence for stealing $1,000 and a five year sentence for stealing $1,000,000, a couple of things might happen. He could say "Yeah, I genuinely believe poor people deserve greater penalties than rich people." But more likely he says "Oh, I guess I was prejudiced." Then if we ask him the same question again, he comes up with two numbers that follow the expected mathematical relationship and punish the greater theft with more jail time.

Bob isn't working off of some predefined algorithm for determining punishment, like "jail time = (10 * amount stolen)/net worth". I don't know if anyone knows exactly what Bob is doing, but at a stab, he's seeing how many unpleasant feelings get generated by imagining the crime, then proposing a jail sentence that activates about an equal amount of unpleasant feelings. If the thought of a homeless man makes images of crime more readily available and so increases the unpleasant feelings, things won't go well for the homeless man. If you're really hungry, that probably won't help either.

So just like nothing automatically synchronizes the intention to study a foreign language and the behavior of studying it, so nothing automatically synchronizes thoughts about punishing the theft of $1000 and punishing the theft of $1000000.

Of course, there is something that non-automatically does it. After all, in order to elicit this strange behavior from Bob, we had to wait until he forgot about the first answer. Otherwise, he would have noticed and quickly adjusted his answers to make sense.

We probably could represent Bob's tendencies as an equation and call it a preference. Maybe it would be a long equation with terms for net worth of criminal, amount stolen, how much food Bob's eaten in the past six hours, and whether his local sports team won the pennant recently, with appropriate coefficients and powers for each. But if Bob saw this equation, he certainly wouldn't endorse it. He'd probably be horrified. It's also unstable: if given a choice, he would undergo brain surgery to remove this equation, thus preventing it from being satisfied. This is why I am reluctant to call these potential formalizations of these equations a "preference".

Instead of saying that Bob has one preference determining his jail time assignments, it would be better to model him as having several tendencies - a tendency to give a certain answer in the $1000 case, a tendency to give a different answer in the $1000000 case, and several tendencies towards things like consistency, fairness, compassion, et cetera.

People strongly consciously endorse these latter tendencies, probably because they're socially useful1. If the Chief of Police says "I know I just put this guy in jail for theft, but I'm going to let this other thief off because he's my friend, and I don't really value consistency that much," then they're not going to stay Chief of Police for very long.

Bayesians and rationalists, in particular, make a big deal out of consistency. One common parable on the importance of consistency is the Dutch Book - a way to get free money from anyone behaving inconsistently. Suppose you have a weighted coin which can land on either heads or tails. There are several good reasons why I should not assign a probability of 66% to heads and 66% to tails, but one of the clearest is this: you can make me a bet that I will give you $2 if it lands on tails and you give me $1 if it lands on heads, and then a second bet where I give you $2 if it lands on heads and you give me $1 if it lands on tails. Whichever way the coin lands, I owe you $1 and you owe me $2 - I have gained a free dollar. So consistency is good if you don't want to be handing dollars out to random people...

...except that the Dutch book itself assumes consistency. If I believe that there is a 66% chance of it landing on heads, but refuse to take a bet at 2:1 odds - or even at 1.5:1 odds even though I should think it's easy money! - then I can't be Dutch booked. I am literally too stupid to be tricked effectively. You would think this wouldn't happen too often, since people would need to construct an accurate mental model to know when they should refuse such a bet, and such an accurate model would tell them they should revise their probabilities - but time after time people have demonstrated the ability to do exactly that.

I have not yet accepted that consistency is always the best course in every situation. For example, in Pascal's Mugging, a random person threatens to take away a zillion units of utility if you don't pay them $5. The probability they can make good on their threat is miniscule, but by multiplying out by the size of the threat, it still ought to motivate you to give the money. Some belief has to give - the belief that multiplication works, the belief that I shouldn't pay the money, or the belief that I should be consistent all the time - and right now, consistency seems like the weakest link in the chain.

The best we can do is seek reflective equilibrium among our tendencies. If you endorse the belief that rich people should not get lighter sentences than poor people more strongly than you endorse the tendency to give the homeless man ten years in jail and the fund manager five, then you can edit the latter tendency and come up with a "fair" sentence. This is Eliezer's defense of reason and philosophy, a powerful justification for morality (see part one here) and it's probably the best we can do in justifying our motivations as well.

Any tendency that has reached reflective equilibrium in your current state is about as close to a preference as you're going to get. It still won't automatically motivate you, of course. But you can motivate yourself toward it obliquely, and come up with the course of action that you most thoroughly endorse.

FOOTNOTES:

1: A tendency toward consistency can cause trouble if someone gains advantage from both of two mutually inconsistent ideas. Trivers' hypothesis predicts that people will consciously deny the inconsistency so they can continue holding both ideas, yet still remain consistent and so socially acceptable. Rationalists are so annoying because we go around telling people they can't do that.

Secrets of the eliminati

93 Post author: Yvain 20 July 2011 10:15AM

Anyone who does not believe mental states are ontologically fundamental - ie anyone who denies the reality of something like a soul - has two choices about where to go next. They can try reducing mental states to smaller components, or they can stop talking about them entirely.

In a utility-maximizing AI, mental states can be reduced to smaller components. The AI will have goals, and those goals, upon closer examination, will be lines in a computer program.

But in the blue-minimizing robot, its "goal" isn't even a line in its program. There's nothing that looks remotely like a goal in its programming, and goals appear only when you make rough generalizations from its behavior in limited cases.

Philosophers are still very much arguing about whether this applies to humans; the two schools call themselves reductionists and eliminativists (with a third school of wishy-washy half-and-half people calling themselves revisionists). Reductionists want to reduce things like goals and preferences to the appropriate neurons in the brain; eliminativists want to prove that humans, like the blue-minimizing robot, don't have anything of the sort until you start looking at high level abstractions.

I took a similar tack asking ksvanhorn's question in yesterday's post - how can you get a more accurate picture of what your true preferences are? I said:

I don't think there are true preferences. In one situation you have one tendency, in another situation you have another tendency, and "preference" is what it looks like when you try to categorize tendencies. But categorization is a passive and not an active process: if every day of the week I eat dinner at 6, I can generalize to say "I prefer to eat dinner at 6", but it would be non-explanatory to say that a preference toward dinner at 6 caused my behavior on each day. I think the best way to salvage preferences is to consider them as tendencies currently in reflective equilibrium.


A more practical example: when people discuss cryonics or anti-aging, the following argument usually comes up in one form or another: if you were in a burning building, you would try pretty hard to get out. Therefore, you must strongly dislike death and want to avoid it. But if you strongly dislike death and want to avoid it, you must be lying when you say you accept death as a natural part of life and think it's crass and selfish to try to cheat the Reaper. And therefore your reluctance to sign up for cryonics violates your own revealed preferences! You must just be trying to signal conformity or something.

The problem is that not signing up for cryonics is also a "revealed preference". "You wouldn't sign up for cryonics, which means you don't really fear death so much, so why bother running from a burning building?" is an equally good argument, although no one except maybe Marcus Aurelius would take it seriously.

Both these arguments assume that somewhere, deep down, there's a utility function with a single term for "death" in it, and all decisions just call upon this particular level of death or anti-death preference.

More explanatory of the way people actually behave is that there's no unified preference for or against death, but rather a set of behaviors. Being in a burning building activates fleeing behavior; contemplating death from old age does not activate cryonics-buying behavior. People guess at their opinions about death by analyzing these behaviors, usually with a bit of signalling thrown in. If they desire consistency - and most people do - maybe they'll change some of their other behaviors to conform to their hypothesized opinion.

One more example. I've previously brought up the case of a rationalist who knows there's no such thing as ghosts, but is still uncomfortable in a haunted house. So does he believe in ghosts or not? If you insist on there being a variable somewhere in his head marked $belief_in_ghosts = (0,1) then it's going to be pretty mysterious when that variable looks like zero when he's talking to the Skeptics Association, and one when he's running away from a creaky staircase at midnight.

But it's not at all mysterious that the thought "I don't believe in ghosts" gets reinforced because it makes him feel intelligent and modern, and staying around a creaky staircase at midnight gets punished because it makes him afraid.

Behaviorism was one of the first and most successful eliminationist theories. I've so far ignored the most modern and exciting eliminationist theory, connectionism, because it involves a lot of math and is very hard to process on an intuitive level. In the next post, I want to try to explain the very basics of connectionism, why it's so exciting, and why it helps justify discussion of behaviorist principles.

Connectionism: Modeling the mind with neural networks

39 Post author: Yvain 19 July 2011 01:16AM

For about a century, people have known that the brain is made up of neurons which connect to each another and perform computations through electrochemical transmission. For about half a century, people have known enough about computers to realize that the brain doesn't look much like one but still computes pretty well regardless. How?

Spreading Activation was one of the first models of mental computation. In this theory, you can imagine the brain as a bunch of nodes in a graph with labels like "Warlord" "Mongolia" "Barbarian", "Genghis Khan" and "Salmon". Each node has certain connections to the others; when they get activated around the same time, it strengthens the connection. When someone asks a question like "Who was that barbaric Mongol warlord, again?" it activates the nodes "warlord", "barbarian", and "Mongol". The activation spreads to all the nodes connected to these, activating them too, and the most strongly activated node will be the one that's closely connected to all three - the barbaric Mongol warlord in question, Genghis Khan. All the while, "salmon", which has no connection to any of these concepts, just sits on its own not being activated. This fits with experience, in which if someone asks us about barbaric Mongol warlords, the name "Genghis Khan" pops into our brain like magic, while we continue to not think about salmon if we weren't thinking about them before.

Bark leash bone wag puppy fetch. If the word "dog" is now running through your head, you may be a victim of spreading activation, as were participants in something called a Deese-Roediger-McDermott experiment, who when asked to quickly memorize a list of words like those and then test their retention several minutes later, were more likely to "remember" "dog" than any of the words actually on the list.

So this does seem attractive, and it does avoid the folk psychology concept of a "belief". The spreading activation network above was able to successfully answer a question without any representation of propositional statements like "Genghis Khan was a barbaric Mongol warlord." And one could get really enthusiastic about this and try to apply it to motivation. Maybe we have nodes like "Hunger", "Food", "McDonalds", and "*GET IN CAR, DRIVE TO MCDONALDS*". The stomach could send a burst of activation to "Hunger", which in turn activates the closely related "Food", which in turn activates the closely related "McDonalds", which in turn activates the closely related "*GET IN CAR, DRIVE TO MCDONALDS*", and then before you know it you're ordering a Big Mac.

But when you try to implement this on a computer, you don't get very far. Although it can perform certain very basic computations, it has trouble correcting itself, handling anything too complicated (the question "name one person who is *not* a barbaric Mongol warlord" would still return "Genghis Khan" on our toy spreading activation network), or making good choices (you can convince the toy network McDonalds is your best dining choice just by saying its name a lot; the network doesn't care about food quality, prices, or anything else.)

This simple spreading activation model also crashes up against modern neuroscience research, which mostly contradicts the idea of a "grandmother cell", ie a single neuron that represents a single concept like your grandmother. Mysteriously, all concepts seem to be represented everywhere at once - Karl Lashley found you can remove any part of a rat's cortex without significantly damaging a specific memory, proving the memory was nonlocalized. How can this be?

Computer research into neural nets developed a model that could answer these and other objects, transforming the immature spreading activation model into full-blown connectionism.

CONNECTIONISM

Connectionism is what happens when you try to implement associationism on a computer and find out it's a lot weirder than you thought.

Take a bunch of miniprocessors called "units" and connect them to each other with unidirectional links. Call some units "inputs" and others "outputs". Decide what you want to do with them: maybe learn to distinguish chairs from non-chairs.

Each unit computes a single value representing its "activity level"; each link has a "strength" with which it links its origin unit to its destination unit. When a unit is "activated" (gets an activity level > 0), it sends that activation along all of its outgoing links. If it has an activation level of .5, and two outgoing links, one to A with strength .33 and one to B with strength -.5, then it sends .165 activation to unit A and -.25 activation to unit B. A and B might also be getting lots of activation from other units they're connected to.

Name your two output units "CHAIR" and "NOT A CHAIR". Connect your many input units to sense-data about the objects you want to classify as chairs or non-chairs; each one could be the luminosity of a pixel in an image of the object, or you could be kind to it and feed it pre-processed input like "IS MADE OF WOOD" and "IS SENTIENT".

Suppose we decide to start with a nice wooden chair. The IS MADE OF WOOD node lights up to its maximum value of 1: it's definitely made of wood! The IS SENTIENT node stays dark; it's definitely not sentient. And then...nothing happens, because we forgot to set the link strengths to anything other than 0. IS MADE OF WOOD is sending activation all over, but it's getting multiplied by zero and everything else stays dark.

We now need an program to train the neural net (or a very dedicated human with lots of free time). The training program knows that the correct answer should have been CHAIR, and so the node we designated "CHAIR" should have lit up. It uses one of several algorithms to change the strengths of the links in such a way that next time the nodes that have currently lit up light up, CHAIR will also light up. For example, it might change the link from IS MADE OF WOOD to CHAIR to .3 (why doesn't it change it all the way to its maximum value? Because that erases all previous data and reduces the system's entire intelligence to what it learned on just this case).

On the other hand, IS SENTIENT is dark, so the training program might infer that IS SENTIENT is not a characteristic of chairs, and change the link strength there accordingly.

The next time the program sees a picture of a wooden chair, IS MADE OF WOOD will light up, and it will send its activation to IS CHAIR, making IS CHAIR light up with .3 units of activation: the program has a weak suspicion that the picture is a chair.

This is a pretty boring neural network, but if we add several hundred input nodes with all conceivable properties relevant to chairhood and spend a lot of computing power, eventually the program will become pretty good at recognizing chairs from nonchairs, and "learn" complicated rules that a three-legged wooden object is a stool which sort of counts as a chair, but a three legged sentient being is an injured dog and sitting on it will only make it angry.

Larger and more complicated neural nets contain "hidden nodes" - the equivalent of interneurons which sit between the input and the output and exist only to perform computations; feedback from an output node to a previous node that can create stable circles of activation, and other complications. They can perform much more difficult classification problems - identifying words from speech, or people from a photograph.

This is interesting because it solves a problem that baffled philosophers for millennia: the difficulty of coming up with good boundaries for categories. Plato famously defined Man as "a featherless biped"; Diogenes famously responded by presenting him with a shaved chicken. There seem to be many soft constraints on humans (can use language, have two legs, have a heartbeat) but there are also examples of humans who violate these constraints (babies, amputees, Dick Cheney) yet still seem obviously human.

Classical computers get bogged down in these problems, but neural nets naturally reason with "cluster structures in thing-space" and are expert classifiers in the same way we ourselves are.

SIMILARITIES BETWEEN NETS AND BRAINS


Even aside from their skill at classifying and pattern-matching, connectionist networks share many properties with brains, such as:

- Obvious structural similarities: neural nets work by lots of units which activate with different strengths and then spread that activation through links; the brain works by lots of neurons which fire at different rates and then spread that activation through axons.

- Lack of a "grandmother cell". A classical computer sticks each bit of memory in a particular location. A neural net stores memories as patterns of activation across all units in the network. In a feedback network, specific oft-repeated patterns can form attractor states to which the network naturally tends if pushed anywhere in the region. Association between one idea and another is not through physical contiguity, but through similarities in the pattern. "Grandmother" probably has most of the same neurons in the same state as "grandfather", and so it takes only a tiny stimulus to push the net from one attractor state to the other.

- Graceful failure: Classical computer programs do not fail gracefully; flip one bit, and the whole thing blows up and you have to spend the rest of your day messing around with a debugger. Destroying a few units in a neural net may only cost it a little bit of its processing power. This matches with the brain: losing a couple of neurons may make you think less clearly; losing a lot of neurons may give you dementia, memory loss and poor judgment. But there's no one neuron without which you just sit there near-catatonic, chanting "ERROR: NEURON 10559020481 NOT RESPONDING." And Karl Lashley can take out any part of a rat's cortex without affecting its memories too much.

- Remembering and forgetting: Neural nets can form memories, and the more the stimulus recurs to them the better they will remember it. But the longer they go without considering the stimulus, the more likely it is that the units involved in the memory-pattern will strengthen other connections, and then it will be harder to get them back in the memory pattern. This is much closer to how humans treat memory than the pristine, eternal encoding of classical computers.

- Ability to quickly locate solutions that best satisfy many soft constraints. What's a good place for dinner that's not too expensive, not more than twenty minutes away, serves decent cocktails, and has burgers for the kids? A classical computer would have to first identify the solution class as "restaurants", then search every restaurant it knows to see if they match each constraint, then fail to return an answer if no such restaurant exists. A neural net will just *settle* on the best answer, and if the cocktails there aren't really that good, it'll just settle but give the answer a lower strength.

- Context-sensitivity. Gold silver copper iron tin, and now when I say "lead", you're thinking of Element 82 (Pb), even though without the context a more natural interpretation is of the "leadership" variety. Currently active units can force others into a different pattern, giving context sensitivity not only to semantic priming as in the above example, but to emotions (people's thoughts follow different patterns when they're happy or sad), situations, and people.

Neural nets have also been used to simulate the results of many popular psychological experiments, including different types of priming, cognitive dissonance, and several of the biases and heuristics.

CONNECTIONISM AND REINFORCEMENT LEARNING

The link between connectionism and associationism is pretty obvious, but the link between connectionism and behaviorism is more elegant.

In most artificial neural nets, you need a training program to teach the net whether it's right or wrong and which way to adjust the weights. Brains don't have that luxury. Instead, part of their training algorithm for cognitive tasks is based on surprise: if you did not expect the sun to rise today, and you saw it rise anyway, you should probably decrease the strength of whatever links led you to that conclusion, and increase the strengths of any links that would have correctly predicted the sunrise.

Motivational links, however, could be modified by reinforcement. If a certain action leads to reward, strengthen the links that led to that action; if it leads to punishment, strengthen the links that would have made you avoid that action.

This explains behaviorist principles as a simple case of connectionism, the one where all the links are nice and straight, and you just have to worry about motivation and not about where cognition is coming from. Many of the animals typically studied by behaviorists were simple enough that this simple case was sufficient.

Although I think connectionism is our best current theory for how the mind works at a low level, it's hard to theorize about just because the networks are so complicated and so hard to simplify. Behaviorism is useful because it reduces the complexity of the networks to a few comprehensible rules, which allow higher level psychological theories and therapies to be derived from them.

Comment author: Nominull 20 July 2011 04:49:26AM 0 points [-]

So how does "not" work, then? It seems like even if you put in a bunch of hidden nodes and distribute the knowledge throughout the brain, you're gonna have trouble with compositional semantics like that.

Comment author: Yvain 20 July 2011 10:04:29AM 0 points [-]

Couldn't "not" negatively reinforce a hidden node level between the input and output?

I'd like to hear what an expert like Phil has to say on this topic.

Comment author: PhilGoetz 19 July 2011 11:49:20PM *  6 points [-]

This simple spreading activation model also crashes up against modern neuroscience research, which mostly contradicts the idea of a "grandmother cell", ie a single neuron that represents a single concept like your grandmother.

...

Association between one idea and another is not through physical contiguity, but through similarities in the pattern. "Grandmother" probably has most of the same neurons in the same state as "grandfather", and so it takes only a tiny stimulus to push the net from one attractor state to the other.

This is extremely unlikely. Associations can be made between concepts long after the patterns for those concepts have been learned. For a different explanation, see my 2000 article, A neuronal basis for the fan effect. It used the idea of convergence zones, promoted by Antonio Damasio (Damasio, A. R. (1990), Synchronous activation in multiple cortical regions: A mechanism for recall. The Neurosciences 2:287–296). My paper did this:

  • Have binary-neuron network 1 represent one concept by a collection of activated nodes
  • Have network 2 (or the same network in the next timestep) represent another concept the same way
  • Have a third network (the convergence zone) learn associations between patterns in those two networks, using the Amari/Hopfield algorithm.

Then the settling of the neurons in the convergence zone into a low-energy state causes the presence of one pattern in network 1 to recall an associated pattern in network 2, with dynamics and error rates that closely mimic John Anderson's experiments on the quantitative measurement of spreading activation in humans.

(I was careful in my article to give credit to Amari, who invented the Hopfield network 10 years before Hopfield did. But I see now the editor "fixed" my reference to no longer give Amari priority.)

Comment author: Yvain 20 July 2011 09:57:52AM 2 points [-]

Thank you.

I know very little about connectionist networks beyond what I have read in a few review articles. I wrote this not because I was the best person to write it but because no one else has written anything on them yet and I had to stumble across a description of them while looking for other stuff, which upset me because I would have loved to have learned about them several years earlier. I would love if you or someone else who is an expert in the field would write something more up-to-date and accurate.

Comment author: PhilGoetz 19 July 2011 11:56:28PM *  3 points [-]

Motivational links, however, could be modified by reinforcement. If a certain action leads to reward, strengthen the links that led to that action; if it leads to punishment, strengthen the links that would have made you avoid that action.

Reward comes along too much later for this to work for humans. Instead, the brain uses temporal difference learning. I no longer remember what was the first, classic paper demonstrating temporal difference error signals in the brain; it may have been A Neural Substrate of Prediction and Reward (1997). Google ("temporal difference learning", brain). "Temporal Difference Models and Reward-Related Learning in the Human Brain" , Neuron, 2003, will be one of the hits.

Comment author: Yvain 20 July 2011 09:54:32AM 0 points [-]

I agree that the brain uses temporal difference learning. I thought temporal difference learning was that reward propagates back to earliest reliable stimulus based on difference between expected and observed, then reinforces it. How is that different from the quoted text except that quoted is simpler and doesn't use that language?

Connectionism: Modeling the mind with neural networks

39 Post author: Yvain 19 July 2011 01:16AM

For about a century, people have known that the brain is made up of neurons which connect to each another and perform computations through electrochemical transmission. For about half a century, people have known enough about computers to realize that the brain doesn't look much like one but still computes pretty well regardless. How?

Spreading Activation was one of the first models of mental computation. In this theory, you can imagine the brain as a bunch of nodes in a graph with labels like "Warlord" "Mongolia" "Barbarian", "Genghis Khan" and "Salmon". Each node has certain connections to the others; when they get activated around the same time, it strengthens the connection. When someone asks a question like "Who was that barbaric Mongol warlord, again?" it activates the nodes "warlord", "barbarian", and "Mongol". The activation spreads to all the nodes connected to these, activating them too, and the most strongly activated node will be the one that's closely connected to all three - the barbaric Mongol warlord in question, Genghis Khan. All the while, "salmon", which has no connection to any of these concepts, just sits on its own not being activated. This fits with experience, in which if someone asks us about barbaric Mongol warlords, the name "Genghis Khan" pops into our brain like magic, while we continue to not think about salmon if we weren't thinking about them before.

Bark leash bone wag puppy fetch. If the word "dog" is now running through your head, you may be a victim of spreading activation, as were participants in something called a Deese-Roediger-McDermott experiment, who when asked to quickly memorize a list of words like those and then test their retention several minutes later, were more likely to "remember" "dog" than any of the words actually on the list.

So this does seem attractive, and it does avoid the folk psychology concept of a "belief". The spreading activation network above was able to successfully answer a question without any representation of propositional statements like "Genghis Khan was a barbaric Mongol warlord." And one could get really enthusiastic about this and try to apply it to motivation. Maybe we have nodes like "Hunger", "Food", "McDonalds", and "*GET IN CAR, DRIVE TO MCDONALDS*". The stomach could send a burst of activation to "Hunger", which in turn activates the closely related "Food", which in turn activates the closely related "McDonalds", which in turn activates the closely related "*GET IN CAR, DRIVE TO MCDONALDS*", and then before you know it you're ordering a Big Mac.

But when you try to implement this on a computer, you don't get very far. Although it can perform certain very basic computations, it has trouble correcting itself, handling anything too complicated (the question "name one person who is *not* a barbaric Mongol warlord" would still return "Genghis Khan" on our toy spreading activation network), or making good choices (you can convince the toy network McDonalds is your best dining choice just by saying its name a lot; the network doesn't care about food quality, prices, or anything else.)

This simple spreading activation model also crashes up against modern neuroscience research, which mostly contradicts the idea of a "grandmother cell", ie a single neuron that represents a single concept like your grandmother. Mysteriously, all concepts seem to be represented everywhere at once - Karl Lashley found you can remove any part of a rat's cortex without significantly damaging a specific memory, proving the memory was nonlocalized. How can this be?

Computer research into neural nets developed a model that could answer these and other objects, transforming the immature spreading activation model into full-blown connectionism.

CONNECTIONISM

Connectionism is what happens when you try to implement associationism on a computer and find out it's a lot weirder than you thought.

Take a bunch of miniprocessors called "units" and connect them to each other with unidirectional links. Call some units "inputs" and others "outputs". Decide what you want to do with them: maybe learn to distinguish chairs from non-chairs.

Each unit computes a single value representing its "activity level"; each link has a "strength" with which it links its origin unit to its destination unit. When a unit is "activated" (gets an activity level > 0), it sends that activation along all of its outgoing links. If it has an activation level of .5, and two outgoing links, one to A with strength .33 and one to B with strength -.5, then it sends .165 activation to unit A and -.25 activation to unit B. A and B might also be getting lots of activation from other units they're connected to.

Name your two output units "CHAIR" and "NOT A CHAIR". Connect your many input units to sense-data about the objects you want to classify as chairs or non-chairs; each one could be the luminosity of a pixel in an image of the object, or you could be kind to it and feed it pre-processed input like "IS MADE OF WOOD" and "IS SENTIENT".

Suppose we decide to start with a nice wooden chair. The IS MADE OF WOOD node lights up to its maximum value of 1: it's definitely made of wood! The IS SENTIENT node stays dark; it's definitely not sentient. And then...nothing happens, because we forgot to set the link strengths to anything other than 0. IS MADE OF WOOD is sending activation all over, but it's getting multiplied by zero and everything else stays dark.

We now need an program to train the neural net (or a very dedicated human with lots of free time). The training program knows that the correct answer should have been CHAIR, and so the node we designated "CHAIR" should have lit up. It uses one of several algorithms to change the strengths of the links in such a way that next time the nodes that have currently lit up light up, CHAIR will also light up. For example, it might change the link from IS MADE OF WOOD to CHAIR to .3 (why doesn't it change it all the way to its maximum value? Because that erases all previous data and reduces the system's entire intelligence to what it learned on just this case).

On the other hand, IS SENTIENT is dark, so the training program might infer that IS SENTIENT is not a characteristic of chairs, and change the link strength there accordingly.

The next time the program sees a picture of a wooden chair, IS MADE OF WOOD will light up, and it will send its activation to IS CHAIR, making IS CHAIR light up with .3 units of activation: the program has a weak suspicion that the picture is a chair.

This is a pretty boring neural network, but if we add several hundred input nodes with all conceivable properties relevant to chairhood and spend a lot of computing power, eventually the program will become pretty good at recognizing chairs from nonchairs, and "learn" complicated rules that a three-legged wooden object is a stool which sort of counts as a chair, but a three legged sentient being is an injured dog and sitting on it will only make it angry.

Larger and more complicated neural nets contain "hidden nodes" - the equivalent of interneurons which sit between the input and the output and exist only to perform computations; feedback from an output node to a previous node that can create stable circles of activation, and other complications. They can perform much more difficult classification problems - identifying words from speech, or people from a photograph.

This is interesting because it solves a problem that baffled philosophers for millennia: the difficulty of coming up with good boundaries for categories. Plato famously defined Man as "a featherless biped"; Diogenes famously responded by presenting him with a shaved chicken. There seem to be many soft constraints on humans (can use language, have two legs, have a heartbeat) but there are also examples of humans who violate these constraints (babies, amputees, Dick Cheney) yet still seem obviously human.

Classical computers get bogged down in these problems, but neural nets naturally reason with "cluster structures in thing-space" and are expert classifiers in the same way we ourselves are.

SIMILARITIES BETWEEN NETS AND BRAINS


Even aside from their skill at classifying and pattern-matching, connectionist networks share many properties with brains, such as:

- Obvious structural similarities: neural nets work by lots of units which activate with different strengths and then spread that activation through links; the brain works by lots of neurons which fire at different rates and then spread that activation through axons.

- Lack of a "grandmother cell". A classical computer sticks each bit of memory in a particular location. A neural net stores memories as patterns of activation across all units in the network. In a feedback network, specific oft-repeated patterns can form attractor states to which the network naturally tends if pushed anywhere in the region. Association between one idea and another is not through physical contiguity, but through similarities in the pattern. "Grandmother" probably has most of the same neurons in the same state as "grandfather", and so it takes only a tiny stimulus to push the net from one attractor state to the other.

- Graceful failure: Classical computer programs do not fail gracefully; flip one bit, and the whole thing blows up and you have to spend the rest of your day messing around with a debugger. Destroying a few units in a neural net may only cost it a little bit of its processing power. This matches with the brain: losing a couple of neurons may make you think less clearly; losing a lot of neurons may give you dementia, memory loss and poor judgment. But there's no one neuron without which you just sit there near-catatonic, chanting "ERROR: NEURON 10559020481 NOT RESPONDING." And Karl Lashley can take out any part of a rat's cortex without affecting its memories too much.

- Remembering and forgetting: Neural nets can form memories, and the more the stimulus recurs to them the better they will remember it. But the longer they go without considering the stimulus, the more likely it is that the units involved in the memory-pattern will strengthen other connections, and then it will be harder to get them back in the memory pattern. This is much closer to how humans treat memory than the pristine, eternal encoding of classical computers.

- Ability to quickly locate solutions that best satisfy many soft constraints. What's a good place for dinner that's not too expensive, not more than twenty minutes away, serves decent cocktails, and has burgers for the kids? A classical computer would have to first identify the solution class as "restaurants", then search every restaurant it knows to see if they match each constraint, then fail to return an answer if no such restaurant exists. A neural net will just *settle* on the best answer, and if the cocktails there aren't really that good, it'll just settle but give the answer a lower strength.

- Context-sensitivity. Gold silver copper iron tin, and now when I say "lead", you're thinking of Element 82 (Pb), even though without the context a more natural interpretation is of the "leadership" variety. Currently active units can force others into a different pattern, giving context sensitivity not only to semantic priming as in the above example, but to emotions (people's thoughts follow different patterns when they're happy or sad), situations, and people.

Neural nets have also been used to simulate the results of many popular psychological experiments, including different types of priming, cognitive dissonance, and several of the biases and heuristics.

CONNECTIONISM AND REINFORCEMENT LEARNING

The link between connectionism and associationism is pretty obvious, but the link between connectionism and behaviorism is more elegant.

In most artificial neural nets, you need a training program to teach the net whether it's right or wrong and which way to adjust the weights. Brains don't have that luxury. Instead, part of their training algorithm for cognitive tasks is based on surprise: if you did not expect the sun to rise today, and you saw it rise anyway, you should probably decrease the strength of whatever links led you to that conclusion, and increase the strengths of any links that would have correctly predicted the sunrise.

Motivational links, however, could be modified by reinforcement. If a certain action leads to reward, strengthen the links that led to that action; if it leads to punishment, strengthen the links that would have made you avoid that action.

This explains behaviorist principles as a simple case of connectionism, the one where all the links are nice and straight, and you just have to worry about motivation and not about where cognition is coming from. Many of the animals typically studied by behaviorists were simple enough that this simple case was sufficient.

Although I think connectionism is our best current theory for how the mind works at a low level, it's hard to theorize about just because the networks are so complicated and so hard to simplify. Behaviorism is useful because it reduces the complexity of the networks to a few comprehensible rules, which allow higher level psychological theories and therapies to be derived from them.

Comment author: jimmy 18 July 2011 07:09:59PM 4 points [-]

If expectation of reward works across domains, how is that different than a goal?

If an addict can use their full intelligence in order to get the reward, then they can't just say "the part of me that wants heroin is too stupid to bargain with".

Comment author: Yvain 19 July 2011 12:35:27AM 2 points [-]

Yeah; I've already admitted I'm confused about that here

Comment author: Bongo 16 July 2011 01:33:57AM *  7 points [-]

Consider the action of making a goal. I go to all my friends and say "Today I shall begin learning Swahili." This is easy to do. There is no chance of me intending to do so and failing; my speech is output by the same processes as my intentions, so I can "trust" it. But this is not just an output of my mental processes, but an input. One of the processes potentially reinforcing my behavior of learning Swahili is "If I don't do this, I'll look stupid in front of my friends."

I know it's only an example but it needs to be pointed out that maybe saying to all your friends that you're going to do it actually makes you less likely to do it.

Comment author: Yvain 18 July 2011 11:50:55PM 7 points [-]

I bet it depends on the condition. I'd anticipate that something very vague like "I will become a writer" would do worse when told to your friends; something very specific like "I'm going to be writing this evening" would do better, especially if the alternative is going out for drinks this evening with your friends and having them ask "Why aren't you writing?"

Comment author: Raemon 17 July 2011 10:37:36PM 4 points [-]

I've actually heard (but not experienced) that working in soup kitchens and similar things actually does make you feel good in a way that isn't obvious until you've tried it. (That's where the whole "warm-fuzzy" charity thing comes from.) So its more of a (- want, + like, + approve) thing.

(It might not work as well for people on lesswrong who've been conditioned to think of it as wasted time that could have been spent raising money for micronutrients to send to third-world countries)

Still, I get the idea.

Comment author: Yvain 18 July 2011 11:49:35PM 1 point [-]

I know it works for some people; unfortunately it doesn't work for me. Though maybe I've just been trying the wrong charitable activities.

Comment author: JGWeissman 18 July 2011 02:58:46AM 18 points [-]

Reductionists want to reduce things like goals and preferences to the appropriate neurons in the brain; eliminativists want to prove that humans, like the blue-minimizing robot, don't have anything of the sort until you start looking at high level abstractions.

Surely you mean that eliminativists take actions which, in their typical contexts, tend to result in proving that humans, like the blue-minimizing robot, don't have anything of the sort until you start looking at high level abstractions.

Comment author: Yvain 18 July 2011 11:46:51PM 7 points [-]

Surely you mean that there are just a bunch of atoms which, when interpreted as a human category, can be grouped together to form a being classifiable as "an eliminativist".

Comment author: Eugine_Nier 17 July 2011 11:15:48PM 6 points [-]

Eliminativism is all well and good if all one wants to do is predict. However, it doesn't help answer questions like "What should I do?", or "What utility function should we give the FAI?"

Comment author: Yvain 18 July 2011 12:12:11AM *  34 points [-]

The same might be said of evolutionary psychology. In which case I would respond that evolutionary psychology helped us stop thinking in a certain stupid way.

Once, we thought that men were attracted to pretty women because there was some inherent property called "beauty", or that people helped their neighbors because there was a universal Moral Law to which all minds would have access. Once it was the height of sophistication to argue whether people were truly good but corrupted by civilization, or truly evil but restrained by civilization.

Evolutionary psychology doesn't answer "What utility function should we give the FAI?", but it gives good reasons to avoid the "solution": 'just tell it to look for the Universal Moral Law accessible to all minds, and then do that.' And I think a lot of philosophy progresses by closing off all possible blind alleys until people grudgingly settle on the truth because they have no other alternative.

I am less confident in my understanding of eliminativism than of evo psych, so I am less willing to speculate on it. But since one common FAI proposal is "find out human preferences, and then do those", if it turns out human preferences don't really exist in a coherent way, that sounds like an important thing to know.

I think many people have alluded to this problem before, and that the people seriously involved in the research don't actually expect it to be that easy, but a clear specification of all the different ways in which it is not quite that easy is still useful. The same is true for "what should I do?"

Comment author: ksvanhorn 17 July 2011 03:53:49AM 5 points [-]

OK, so you've given us the bad news. Is there any good news, i.e., research showing how you can get a more accurate picture of what your true preferences are and the true reasons for your behaviors?

Comment author: Yvain 17 July 2011 09:45:16PM 12 points [-]

I don't think there are true preferences. In one situation you have one tendency, in another situation you have another tendency, and "preference" is what it looks like when you try to categorize tendencies. But categorization is a passive and not an active process: if every day of the week I eat dinner at 6, I can generalize to say "I prefer to eat dinner at 6", but it would be false to say that some stable preference toward dinner at 6 is what causes my behavior on each day.

I think the best way to salvage preferences is to consider them as tendencies in reflective equilibrium. I'll explain that more later.

Approving reinforces low-effort behaviors

91 Yvain 17 July 2011 08:43PM

In addition to "liking" to describe pleasure and "wanting" to describe motivation, we add "approving" to describe thoughts that are ego syntonic.

A heroin addict likes heroin. He certainly wants more heroin. But he may not approve of taking heroin. In fact, there are enough different cases to fill in all eight boxes of the implied 2x2x2 grid (your mileage may vary):

+wanting/+liking/+approving: Romantic love. If you're doing it right, you enjoy being with your partner, you're motivated to spend time with your partner, and you think love is a wonderful (maybe even many-splendored) thing.

+wanting/+liking/-approving: The aforementioned heroin addict feels good when taking heroin, is motivated to get more, but wishes he wasn't addicted.

+wanting/-liking/+approving: I have taken up disc golf. I play it every day, and when events conspire to prevent me from playing it, I seethe. I approve of this pastime: I need to take up more sports, and it helps me spend time with my family. But when I am playing, all I feel is stressed and angry that I was literally *that* close how could I miss that shot aaaaarggghh.

+wanting/-liking/-approving: The jaded addict. I have a friend who says she no longer even enjoys coffee or gets any boost from it, she just feels like she has to have it when she gets up.

-wanting/+liking/+approving: Reading non-fiction. I enjoy it when I'm doing it, I think it's great because it makes me more educated, but I can rarely bring myself to do it.

-wanting/-liking/+approving:
Working in a soup kitchen. Unless you're the type for whom helping others is literally its own reward it's not the most fun thing in the world, nor is it the most attractive, but it makes you a Good Person and so you should do it.

-wanting/+liking/-approving:
The non-addict. I don't want heroin right now. I think heroin use is repugnant. But if I took some, I sure bet I'd like it.

-wanting/-liking/-approving:
Torture. I don't want to be tortured, I wouldn't like it if I were, and I will go on record declaring myself to be against it.


Discussion of goals is mostly about approving; a goal is an ego-syntonic thought. When we speak of goals that are hard to achieve, we're usually talking about +approving/-wanting. The previous discussion of learning Swahili is one example; more noble causes like Working To Help The Less Fortunate can be others.

Ego syntonicity itself is mildly reinforcing by promoting positive self-image. Most people interested in philosophy have at least once sat down and moved their arm from side to side, just to note that their mind really does control their body; the mental processes that produced curiosity about philosophy were sufficiently powerful to produce that behavior as well. Some processes, like moving one's arm, or speaking aloud, or engaging in verbal thought, are so effortless, and so empty of other reinforcement either way, that we usually expect them to be completely under the control of the mild reinforcement provided by approving of those behaviors.

Other behaviors take more effort, and are subject not only to discounting but to many other forms of reinforcement. Unlike the first class of behaviors, we expect to experience akrasia when dealing with this latter sort. This offers another approach to willpower: taking low-effort approving-influenced actions that affect the harder road ahead.

Consider the action of making a goal. I go to all my friends and say "Today I shall begin learning Swahili." This is easy to do. There is no chance of me intending to do so and failing; my speech is output by the same processes as my intentions, so I can "trust" it. But this is not just an output of my mental processes, but an input. One of the processes potentially reinforcing my behavior of learning Swahili is "If I don't do this, I'll look stupid in front of my friends."

Will it be enough? Maybe not. But this is still an impressive process: my mind has deliberately tweaked its own inputs to change the output of its own algorithm. It's not even pretending to be working off of fixed preferences anymore, it's assuming that one sort of action (speaking) will work differently from another action (studying), because the first can be executed solely through the power of ego syntonicity, and the second may require stronger forms of reinforcement. It gets even weirder when goals are entirely mental: held under threat not of social disapproval, but of feeling bad because you're not as effective as you thought. The mind is using mind's opinion of the mind to blackmail the mind.

But we do this sort of thing all the time. The dieter who successfully avoids buying sweets when he's at the store because he knows he would eat them at home is changing his decisions by forcing effort discounting of any future sweet-related reward (because he'd have to go back to the store). The binge shopper who freezes her credit cards in a block of ice is using time discounting in the same way. The rationalist who sends money to stickk is imposing a punishment with a few immediate and effortless mouse clicks. Even the poor unhappy person who tries to conquer through willpower alone is trying to set up the goal as a Big Deal so she will feel extra bad if she fails. All are using their near-complete control of effortless immediate actions to make up for their incomplete control of high-effort long-term actions.

This process is especially important to transhumanists. In the future, we may have the ability to self-modify in complicated ways that have not built up strong patterns of reinforcement around them. For example, we may be able to program ourselves at the push of a button. Such programming would be so effortless and empty of past reinforcement that behavior involving it would be reinforced entirely by our ego-syntonic thoughts. It would supersede our current psychodynamics, in which our thoughts are only tenuously linked to our important actions and major life decisions. A Singularity in which behaviors were executed by effectively omnipotent machines that acted on our preferences - preferences which we would presumably communicate through low-effort channels like typed commands - would be an ultimate triumph for the ego-syntonic faction of the brain.

Ego syntonic thoughts and values

53 Yvain 17 July 2011 08:43PM

Related to: Will your real preferences please stand up?

Last week I read a book in which two friends - let's call them John and Lisa so I don't spoil the book for anyone who wanders into it - got poisoned. They only had enough antidote for one person and had to decide who lived and who died. John, who was much larger than Lisa, decided to hold Lisa down and force the antidote down her throat. Lisa just smirked; she'd replaced the antidote with a lookalike after slipping the real thing into John's drink earlier in the day.

These are good friends. Not only was each willing to give the antidote to the other, but each realized it would be unfair to make the other live with the crippling guilt of having chosen to survive at the expense of a friend's life, and so decided to force the antidote on the other unwillingly to prevent any guilt over the fateful decision. Whatever you think of the ethics of their decision, you can't help admire the thought processes.

Your brain might be this kind of a friend.

In Trivers' hypothesis of self-deception, one of the most important functions of the conscious mind is effective signaling. Since people have the potential to be excellent lie-detectors, the conscious mind isn't given full access to information so that it can lend the ring of truth to useful falsehoods.

But this doesn't always work. If you're addicted to heroin, at some point you're going to notice. And telling your friends "No, I'm not addicted, it's just a coincidence that I take heroin every day," isn't going to cut it. But there's another way in which the brain can sequester information to promote effective signaling.

Wikipedia defines the term "ego syntonic" as "referring to behaviors, values, feelings that are in harmony with or acceptable to the needs and goals of the ego, or consistent with one's ideal self-image", and "ego dystonic" as the opposite of that. A heroin addict might say "I hate heroin, but somehow I just feel compelled to keep taking it." But an astronaut will say "I love being an astronaut and I worked hard to get into this career."

Both the addict and the astronaut have desires: the addict wants to take heroin, the astronaut wants to fly in space. But the addict's desires manifest as an unpleasant compulsion from outside, and the astronaut's manifest as a genuine and heartfelt love.

Suppose that in the original example, John predicted that Lisa would ask for the antidote, but later feel guilty about it and believe she was a bad person. By presenting the antidote to Lisa in the form of an external compulsion, he allows Lisa to do what she wanted anyway and avoid the associated guilt.

Under Trivers' hypothesis, the compulsion for heroin works the same way. The heroin addict's definitely going to get that heroin, but by presenting the desire in the form of an external compulsion, the unconscious saves the heroin addict from the social stigma of "choosing" heroin. This allows the addict to create a much more sympathetic narrative than the alternative: "I want to support my family and keep clean, but for some reason these compulsions keep attacking me," instead of "Yeah, I like heroin more than I like supporting my family. Deal with it."

EGO SYNTONIA, DYSTONIA, AND WILLPOWER

Willpower cashes out as the action of ego syntonic thoughts and desires against ego dystonic thoughts and desires.

The aforementioned heroin addict may have several reinforcers both promoting and discouraging heroin use. On the plus side, heroin itself is very strongly rewarding. On the minus, it can lead to both predicted and experienced poverty, loss of friendships, loss of health, and death.

Worrying about the latter factors determining heroin use - the factors that make heroin a bad idea - is socially encouraged and good signaling material. A person wanting to put their best face forward should believe themselves to be the sort of person who cares about these things. These desires will be ego syntonic. Wanting to take heroin, on the other hand, is a socially unacceptable desire, so it presents as dystonic.

If the latter syntonic factors win out over the dystonic factors, this feels from the inside like "I exerted willpower and managed to overcome my heroin addiction." If the dystonic factors win out over the syntonic factors, this feels from the inside like "I didn't have enough willpower to overcome my heroin addiction."

DYSTONIC DESIRES IN ABNORMAL PSYCHOLOGY

There is some speculation that the brain has one last trick up its sleeve to deal with desires that are so unpleasant and unacceptable that even manifesting them as external compulsions isn't good enough: it splits them off into weird alternate personalities.

One of the classic stereotypes of the insane is that they hear voices telling them to kill people. During my short time working at a psychiatric hospital, I was surprised by how spot-on this stereotype was: meeting someone who heard voices telling him to kill people was an almost daily occurrence. Other voices would have other messages: maybe that the patient was a horrible person who deserved to die, or that the patient must complete some bizarre ritual or else doom everybody. There were relatively fewer voices saying "Hey, let's go fishing!"

One theory explaining these voices is that they are an extreme reaction to highly ego dystonic thoughts. Some aspect of the patients' mental disease gives them obsessive thoughts about (though rarely a desire for) killing people. Genuinely wanting to kill people would make you a bad person, but even saying "I feel a strong compulsion to kill people" is pretty bad too. The best the brain can do with this desire is pitch it as a completely different person by presenting it as an outside voice speaking to the patient.

Although everything about dissociative identity disorder (aka multiple personality disorder) is controversial including its very existence, perhaps one could sketch a similar theory explaining that condition in the same framework of separating out dystonic thoughts.

SUMMARY

A conscious/unconscious divide helps signaling by allowing the conscious mind to hold only socially acceptable beliefs, which it can broadcast without detectable falsehood. Socially acceptable ideas present as the conscious mind's own beliefs and desires; unacceptable ones present as compulsions from afar. The balance of ego syntonic and dystonic desires presents as willpower. In extreme cases, some desires may be so ego dystonic that they present as external voices.

Comment author: Will_Newsome 16 July 2011 10:39:46PM *  4 points [-]

That kind of seems to be a red herring, though, in the absence of such buttons. It's an empirical matter and one that is worth paying close attention to, but it seems to me that for a large number of people the ego, the super-ego, the pre-frontal cortex, and the many shards of Azathoth are all different and difficult to differentiate. I know that I internally use the notion "I [egosyntonicly and upon reflection] want to do X" ("I desire taking action X or set of actions X", "We, the spaciotemporal coalition of mind fragments who are currently reflecting, want to do X") when I at least partially mean "I want to signal virtue Y", "I want to be seen as a person who does things like X", "I am afraid of being seen as a person who doesn't do X", "I am afraid of being seen as a person who doesn't possess virtue Y", "I am afraid of being seen as someone who doesn't believe virtue Y is desirable", "I am afraid of not doing X", "I want to believe that I want to do X", "I am afraid of the consequences of not believing that I want to do X", et cetera ad nauseum. (Same for "I want to be (adjective)", "I want to be a (adjectival noun)", "I want to possess (concrete or abstract noun)", et cetera.)

ETA: (I use "am afraid of" where perhaps I should use "find aversive", the latter being more general and more accurate. Fear is a similar but narrower phenomenon, I think, more Near and less Far than the most common kinds of aversion.)

The point being that each of those interpretations of my "want" emphasizes a mechanistically and possibly neuroanatomically different source of attraction and/or aversion, the conglomeration of which is difficult to break down into pieces and thus difficult to analyze to determine the 'biggest' causal factors therein. It is unclear to me whether or not the Pareto principle applies to the analysis of the sources of egosyntonic aversion/attraction, and it is also unclear if empirical introspection is enough to truthfully identify the biggest causal factor in the event that the Pareto principle does in fact apply.

EATA: I remember thinking that a modernized and skillfully interpreted version of Jungian psychology would be useful for doing this kind of introspection.

Comment author: Yvain 17 July 2011 12:17:12PM 3 points [-]

You're right; I concede that my model is too simplistic. I'll have to think about it further.

Comment author: Yvain 17 July 2011 12:01:30PM *  36 points [-]

With apologies for not commenting more on your other points until I've read the thread more closely:

I suggest avoiding changing your estimate of a person's character, or assuming bad faith, just because "someone compared X to Y".

I don't know if there's a formal name for this fallacy, but it never fails that when someone compares A to B regarding characteristic Y, someone else interprets them as implying A is also comparable to B regarding characteristic Z.

I can't think of a real example at this hour of the night, but I did read a post a while ago where a mathematician tried to model the memetic spread of popular religions like Mormonism using equations from epidemiology. If you treat the religion as a disease which gets spread from a "case zero" to their close contacts and so on to their close contacts, then maybe you can use epidemiology to predict how quickly the religion spreads. I don't remember if it worked or not but it was a clever idea.

But imagine some Mormon reading that and saying "Atheist mathematicians at liberal universities are writing papers explicitly comparing Mormons to bacteria now. I guess the next step is to recommend we get eradicated to 'cure' the 'disease'".

(sorry to pick on the Mormons, it was the first example that popped into my head)

Yeah, they're comparing Mormonism to a disease, but only along one limited axis, not in general, and not in a way that implies what the objector thinks it implies.

I don't trust anyone including myself to avoid this, so I try to avoid accusatory "he's comparing X to Y!" statements. If someone is really digging themselves into a hole - if they say something like "Jews should be eliminated like vermin" - then you should just say "he said Jews should be eliminated like vermin!" and not the weaker "he compared Jews to vermin". Yeah, occasionally you make a type ii error - if Hitler says "Jews are as common here as cockroaches" then he's trying to imply something beyond just numbers - but usually you have more evidence against those sorts of people than just one comparative statement.

Comment author: Jordan 16 July 2011 11:50:24PM 3 points [-]

Great post, great review of the literature.

Where do you get most of your references? Do you wade through the literature, or do you use review papers? I'd love to see a book length compilation with the same density as this post.

Comment author: Yvain 17 July 2011 03:31:42AM 1 point [-]

With one or two exceptions, these were all taken from the link "Verbal Reports on Mental Processes" at the beginning of the post.

Comment author: KPier 16 July 2011 11:57:08PM 11 points [-]

I've noticed on your last posts that most of the studies cited are decades old; is that because this is considered a settled question in behavioral science, because a lot of these experiments wouldn't pass modern ethics standards, or something else?

typo:

when the subject was holding on two one cord ey couldn't reach the other.

Comment author: Yvain 17 July 2011 03:28:10AM *  15 points [-]

It's because I am lazy enough that I took all of these from a single excellent review article on the subject written in the late 1970s. As far as I know, the research since then has confirmed the same points.

Fixed the typo, and thank you, but I find the mental processes generating it interesting. I used "two" instead of "to" right before the number "one" - my guess is that my being about to write "one" semantically primed my concept of "number", making me write "to" as "two". If I could think of a way to search, I'd love to see how many of the same two/to error on the Internet occur right around mention of a number.

The limits of introspection

56 Yvain 16 July 2011 09:00PM

Related to: Inferring Our Desires

The last post in this series suggested that we make up goals and preference for other people as we go along, but ended with the suggestion that we do the same for ourselves. This deserves some evidence.

One of the most famous sets of investigations into this issue was Nisbett and Wilson's Verbal Reports on Mental Processes, the discovery of which I owe to another Less Wronger even though I can't remember who. The abstract says it all:

When people attempt to report on their cognitive processes, that is, on the processes mediating the effects of a stimulus on a response, they do not do so on the basis of any true introspection. Instead, their reports are based on a priori, implicit casual theories, or judgments about the extent to which a particular stimulus is a plausible cause of a given response. This suggests that though people may not be able to observe directly their cognitive processes, they will sometimes be able to report accurately about them. Accurate reports will occur when influential stimuli are salient and are plausible causes of the responses they produce, and will not occur when stimuli are not salient or are not plausible causes.

In short, people guess, and sometimes they get lucky. But where's the evidence?

Nisbett & Schachter, 1966. People were asked to get electric shocks to see how much shock they could stand (I myself would have waited to see if one of those see-how-much-free-candy-you'll-eat studies from the post last week was still open). Half the subjects were also given a placebo pill which they were told would cause heart palpitations, tremors, and breathing irregularities - the main problems people report when they get shocked. The hypothesis: people who took the pill would attribute much of the unpleasantness of the shock to the pill instead, and so tolerate more shock. This occurred right on schedule: people who took the pill tolerated four times as strong a shock as controls. When asked why they did so well, the twelve subjects in the experimental group came up with fabricated reasons; one example given was "I played with radios as a child, so I'm used to electricity." Only three of twelve subjects made a connection between the pill and their shock tolerance; when the researchers revealed the deception and their hypothesis, most subjects said it was an interesting idea and probably explained the other subjects, but it hadn't affected them personally.

Zimbardo et al, 1965. Participants in this experiment were probably pleased to learn there were no electric shocks involved, right up until the point where the researchers told them they had to eat bugs. In one condition, a friendly and polite researcher made the request; in another, a surly and arrogant researcher asked. Everyone ate the bug (experimenters can be pretty convincing), but only the group accosted by the unpleasant researcher claimed to have liked it. This confirmed the team's hypothesis: the nice-researcher group would know why they ate the bug - to please their new best friend - but the mean-researcher group would either have to admit it was because they're pushovers, or explain it by saying they liked eating bugs. When asked after the experiment why they were so willing to eat the bug, they said things like "Oh, it's just one bug, it's no big deal." When presented with the idea of cognitive dissonance, they once again agreed it was an interesting idea that probably affected some of the other subjects but of course not them.

Maier, 1931. Subjects were placed in a room with several interesting tools and asked to come up with as many solutions as possible to a puzzle about tying two cords together. One end of each cord was tied to the ceiling, and when the subject was holding on to one cord they couldn't reach the other. A few solutions were obvious, such as tying an extension cord to each, but the experiment involved a more complicated solution - tying a weight to a cord and using it as a pendulum to bring it into reach of the other. Subjects were generally unable to come up with this idea on their own in any reasonable amount of time, but when the experimenter, supposedly in the process of observing the subject, "accidentally" brushed up against one cord and set it swinging, most subjects were able to develop the solution within 45 seconds. However, when the experimenter asked immediately afterwards how they came up with the pendulum idea, the subjects were completely unable to recognize the experimenter's movement as the cue, and instead came up with completely unrelated ideas and invented thought processes, some rather complicated. After what the study calls "persistent probing", less than a third of the subjects mentioned the role of the experimenter.

Latane & Darley, 1970. This is the famous "bystander effect", where people are less likely to help when there are others present. The researchers asked subjects in bystander effect studies what factors influenced their decision not to help; the subjects gave many, but didn't mention the presence of other people.

Nisbett & Wilson, 1977. Subjects were primed with lists of words all relating to an unlisted word (eg "ocean" and "moon" to elicit "tide"), and then asked the name of a question, one possible answer to which involved the unlisted word (eg "What's your favorite detergent?" "Tide!"). The experimenters confirmed that many more people who had been primed with the lists gave the unlisted answer than control subjects (eg more people who had memorized "ocean" and "moon" gave Tide as their favorite detergent). Then they asked subjects why they had chosen their answer, and the subjects generally gave totally unrelated responses (eg "I love the color of the Tide box" or "My mother uses Tide"). When the experiment was explained to subjects, only a third admitted that the words might have affected their answer; the rest kept insisting that Tide was really their favorite. Then they repeated the process with several other words and questions, continuing to ask if the word lists influenced answer choice. The subjects' answers were effectively random - sometimes they believed the words didn't affect them when statistically they probably did, other times they believed the words did affect them when statistically they probably didn't.

Nisbett & Wilson, 1977. Subjects in a department store were asked to evaluate different articles of clothing in a line. As usually happens in this sort of task, people disproportionately chose the rightmost object (four times as often as the leftmost), no matter which object was on the right; this is technically referred to as a "position effect". The customers were asked to justify their choices and were happy to do so based on different qualities of the fabric et cetera; none said their choice had anything to do with position, and the experimenters dryly mention that when they asked the subjects if this was a possibility, "virtually all subjects denied it, usually with a worried glance at the interviewer suggesting they felt that they...were dealing with a madman".

Nisbett & Wilson, 1977. Subjects watched a video of a teacher with a foreign accent. In one group, the video showed the teacher acting kindly toward his students; in the other, it showed the teacher being strict and unfair. Subjects were asked to rate how much they liked the teacher, and also how much they liked his appearance and accent, which were the same across both groups. Because of the halo effect, students who saw the teacher acting nice thought he was attractive with a charming accent; people who saw the teacher acting mean thought he was ugly with a harsh accent. Then subjects were asked whether how much they liked the teacher had affected how much they liked the appearance and accent. They generally denied any halo effect, and in fact often insisted that part of the reason they hated the teacher so much was his awful clothes and annoying accent - the same clothes and accent which the nice-teacher group said were part of the reason they liked him so much!

There are about twice as many studies listed in the review article itself, but the trend is probably getting pretty clear. In some studies, like the bug-eating experiment, people perform behaviors and, when asked why they performed the behavior, guess wrong. Their true reasons for the behavior are unclear to them. In others, like the clothes position study, people make a choice, and when asked what preferences caused the choice, guess wrong. Again, their true reasons are unclear to them.

Nisbett and Wilson add that when they ask people to predict how they would react to the situations in their experiments, people "make predictions that in every case were similar to the erroneous reports given by the actual subjects." In the bystander effect experiment, outsiders predict the presence or absence of others wouldn't affect their ability to help, and subjects claim (wrongly) that the presence or absence of others didn't affect their ability to help.

In fact, it goes further than this. In the word-priming study (remember? The one with Tide detergent?) Nisbett and Wilson asked outsiders to predict which sets of words would change answers to which questions (would hearing "ocean" and "moon" make you pick Tide as your favorite detergent? Would hearing "Thanksgiving" make you pick Turkey as a vacation destination?). The outsiders' guesses correlated not at all with which words genuinely changed answers, but very much with which words the subjects guessed had changed their answers. Perhaps the subjects' answers looked a lot like the outsiders' answers because both were engaged in the same process: guessing blindly.

These studies suggest that people do not have introspective awareness to the processes that generate their behavior. They guess their preferences, justifications, and beliefs by inferring the most plausible rationale for their observed behavior, but are unable to make these guesses qualitatively better than outside observers. This supports the view presented in the last few posts: that mental processes are the results of opaque preferences, and that our own "introspected" goals and preferences are a product of the same machinery that infers goals and preferences in others in order to predict their behavior.

Comment author: jimmy 16 July 2011 12:56:46AM 4 points [-]

Who says your "unconscious" is stupid and doesn't have reflectively consistent preferences? Certain parts, yes. Others, no.

Hell, there are people with entire alternate personalities that they aren't aware of living inside their heads. You've been using addicts as an example for a 'ego dystonic' urge that isn't a preference. Why then do addicts have such a hard time checking themselves into rehab?

Comment author: Yvain 16 July 2011 08:55:16PM 3 points [-]

If DID is a real disorder, the stable personality/preference thing seems to be one of the factors that differentiates sufferers from neurotypical people.

Addicts have such a hard time checking themselves into rehab because behaviors aren't based on "preferences", they're based on expectation of reward. Rehab means no drug use for a long time (unpleasant), probable social status hit, and only later a non-addicted state (pleasant but heavily time-discounted).

Comment author: Nominull 16 July 2011 01:24:14AM 9 points [-]

I think you overstate the extent to which one's ego dystonic desires don't try to steer the universe. If I want heroin, I'm not going to limit myself to previously reinforced means of acquiring heroin, I'll come up with creative new heroin acquisition strategies.

Comment author: Yvain 16 July 2011 08:50:39PM *  8 points [-]

If I wanted peace on earth, I would not be willing to press a button that would eliminate my desire for peace on earth. But if I "wanted" heroin, I would be willing to press a button that would eliminate my desire for heroin, and then consider the problem solved.

Comment author: Kaj_Sotala 16 July 2011 05:53:17PM *  6 points [-]

"My thoughts, not to mention these words I am typing, are effortless and immediate, and so allied with the conscious faction of my mind. We intend to respect that alliance by believing that the conscious mind is the best, and by trying to convince you of this as well."

I'm not at all sure conscious/unconscious is good terminology to use here.

For one, there isn't a single unified unconscious, but a vast array of different not-consciously-accessible modules. Neither is there a single unified consciousness, for that matter. A module can be either conscious or unconscious, depending on whether it happens to be active at the moment in question. Trivial examples: depending on whether you happen to be paying attention to your own thoughts or the external world, objects in your visual field may or may not be conscious. Things like annoyance towards something that somebody did may be either active and conscious (when you're annoyed with them in general), or dormant and unconscious (when you're mostly thinking of how great they are). Various desires or wants may be tugging at you at an unconscious level until they reach a conscious level, and so on.

Furthermore, there's the fact that all the processes that actually select the thoughts that are promoted to conscious awareness are themselves unconscious. All the skills you might employ on to make your decisions are sufficeintly automated that they for the most part operate on an unconscious level, only returning you the results of their analyses. The parts of your knowledge store that are activated and tagged as relevant for this task are again chosen by unconscious processes. Et cetera, et cetera.

You could try to use the phrase "allied with your consciousness" here, to include the parts of your unconscious that are helping out your consciousness... but then again, which consciousness? Consciousness is just a generic label for the modules and processes that happen to be active "in a conscious manner" at a certain point of time. And don't forget that we employ different kinds of processing depending on our mood, too.

There's also the problem that for e.g. our consciously held ethics are just an imperfect model built on the intuitive moral judgments our unconscious outputs. Our conscious mind observes some of its own reactions to something, postulates some formal ethical principles and tries them out, until eventually some annoying philosopher comes up with something like the Repugnant Conclusion. Then our unconscious outputs a negative reaction, showing our ethics wasn't good enough after all, and then we seek to rationalize this judgment with a revamped ethical system. In other words, our unconscious knows our ethics better than our conscious mind does: our conscious mind is just making guesses based on what the unconscious mind says. (Or to be more specific, unconscious processes generate guesses, some of which are given to our conscious mind to evaluate.) Yes, occasionally the conscious mind says something like "my version is better, shut up", but often we do end up accepting the judgment of the unconscious mind.

Comment author: Yvain 16 July 2011 08:45:13PM 1 point [-]

I agree with you about terminology. Ego syntonic vs. ego dystonic desires is probably a better way to put it.

Comment author: Will_Sawin 16 July 2011 02:20:10PM 16 points [-]

Man will you look silly if Barack Obama nukes Toronto today.

Comment author: Yvain 16 July 2011 08:42:36PM *  4 points [-]

1:35 PM and still in the clear!

To what degree do we have goals?

45 Yvain 15 July 2011 11:11PM

Related: Three Fallacies of Teleology

NO NEGOTIATION WITH UNCONSCIOUS

Back when I was younger and stupider, I discussed some points similar to the ones raised in yesterday's post in Will Your Real Preferences Please Stand Up. I ended it with what I thought was the innocuous sentences "Conscious minds are potentially rational, informed by morality, and qualia-laden. Unconscious minds aren't, so who cares what they think?"

A whole bunch of people, including no less a figure than Robin Hanson, came out strongly against this, saying it was biased against the unconscious mind and that the "fair" solution was to negotiate a fair compromise between conscious and unconscious interests.

I continue to believe my previous statement - that we should keep gunning for conscious interests and that the unconscious is not worthy of special consideration, although I think I would phrase it differently now. It would be something along the lines of "My thoughts, not to mention these words I am typing, are effortless and immediate, and so allied with the conscious faction of my mind. We intend to respect that alliance by believing that the conscious mind is the best, and by trying to convince you of this as well." So here goes.

It is a cardinal rule of negotiation, right up there with "never make the first offer" and "always start high", that you should generally try to negotiate only with intelligent beings. Although a deal in which we offered tornadoes several conveniently located Potemkin villages to destroy and they agreed in exchange to limit their activity to that area would benefit both sides, tornadoes make poor negotiating partners.

Just so, the unconscious makes a poor negotiating partner. Is the concept of "negotiation" a stimulus, a reinforcement, or a behavior? No? Then the unconscious doesn't care. It's not going to keep its side of any "deal" you assume you've made, it's not going to thank you for making a deal, it's just going to continue seeking reward and avoiding punishment.

This is not to say people should repress all unconscious desires as strongly as possible. Overzealous attempts to control wildfires only lead to the wildfires being much worse when they finally do break out, because they have more unburnt fuel to work with. Modern fire prevention efforts have focused on allowing controlled burns, and the new focus has been successful. But this is because of an understanding of the mechanisms determining fire size, not because we want to be fair to the fires by allowing them to burn at least a little bit of our land.

One difference between wildfires and tornadoes on one hand, and potential negotiating partners on the other, is that the partners are anthropomorphic; we model them as having stable and consistent preferences that determine their actions. The tornado example above was silly not only because it imagining tornadoes sitting down to peace talks, but because it assumed their demand in such peace talks would be more towns to destroy. Tornadoes do destroy towns, but they don't want to. That's just where the weather brings them. It's not even just a matter of how they don't hit towns any more than chance; even if some weather pattern (maybe something like the heat island effect) always drove tornadoes inexorably to towns, they wouldn't *want* to destroy towns, it would just be a consequences of the meteorological laws that they followed.

Eliezer described the Blue-Minimizing Robot by saying "it doesn't seem to steer the universe any particular place, across changes of context". In some reinforcement learning paradigms, the unconscious behaves the same way. If there is a cookie in front of me and I am on a diet, I may feel an ego dystonic temptation to eat the cookie - one someone might attribute to the "unconscious". But this isn't a preference - there's not some lobe of my brain trying to steer the universe into a state where cookies get eaten. If there were no cookie in front of me, but a red button that teleported one cookie from the store to my stomach, I would have no urge whatsoever to press the button; if there were a green button that removed the urge to eat cookies, I would feel no hesitation in pressing it, even though that would steer away from the state in which cookies get eaten. If you took the cookie away, and then distracted me so I forgot all about it, when I remembered it later I wouldn't get upset that your action had decreased the number of cookies eaten by me. The urge to eat cookies is not stable across changes of context, so it's just an urge, not a preference.

Compare an ego syntonic goal like becoming an astronaut. If there were a button in front of little Timmy who wants to be an astronaut when he grows up, and pressing the button would turn him into an astronaut, he'd press it. If there were a button that would remove his desire to become an astronaut, he would avoid pressing it, because then he wouldn't become an astronaut. If I distracted him and he missed the applications to astronaut school, he'd be angry later. Ego syntonic goals behave to some degree as genuine preferences.

This is one reason I would classify negotiating with the unconscious in the same category as negotiating with wildfires and tornadoes: it has tendencies and not preferences.

The conscious mind does a little better. It clearly understands the idea of a preference. To the small degree that its "approving" or "endorsing" function can motivate behavior, it even sort of acts on the preference. But its preferences seem divorced from the reality of daily life; the person who believes helping others is the most important thing, but gives much less than half their income to charity, is only the most obvious sort of example.

Where does this idea of preference come from, and where does it go wrong?

WHY WE MODEL OTHERS WITH GOALS

In The Blue Minimizing Robot, observers mistakenly interpreted a robot with a simple program about when to shoot its laser as being a goal-directed agent. Why?

This isn't an isolated incident. Uneducated people assign goal-directed behavior to all sorts of phenomena. Why do rivers flow downhill? Because water wants to reach the lowest level possible. Educated people can be just as bad, even when they have the decency to feel a little guilty about it. Why do porcupines have quills? Evolution wanted them to resist predators. Why does your heart speed up when you exercise? It wants to be able to provide more blood to the body.

Neither rivers nor evolution nor the heart are intelligent agents with goal-directed behavior. Rivers behave in accordance with the laws of gravity when applied to uneven terrain. Evolution behaves in accordance with the biology of gene replication, not to mention common-sense ideas about things that replicate becoming more common. And the heart blindly executes adaptations built into it during its evolutionary history. All are behavior-executors and not utility-maximizers.

An intelligent computer program provides a more interesting example of a behavior executor. Consider the AI of a computer game - Civilization IV, for instance. I haven't seen it, but I imagine it's thousands or millions of lines of code which when executed form a viable Civilization strategy.

Even if I had open access to the Civilization IV AI source code, I doubt I could fully understand it at my level. And even if I could fully understand it, I would never be able to compute the AI's likely next move by hand in a reasonable amount of time. But I still play Civilization IV against the AI, and I'm pretty good at predicting its movements. Why?

Because I model the AI as a utility-maximizing agent that wants to win the game. Even though I don't know the algorithm it uses to decide when to attack a city, I know it is more likely to win the game if it conquers cities - so I can predict that leaving a city undefended right on the border would be a bad idea. Even though I don't know its unit selection algorithm, I know it will win the game if and only if its units defeat mine - so I know that if I make an army with disproportionately many mounted units, I can expect the AI to build lots of pikemen.

I can't predict the AI by modeling the execution of its code, but I can predict the AI by modeling the achievements of its goals.

The same situation is true of other human beings. What will Barack Obama do tomorrow? If I try to consider the neural network of his brain, the position of each synapse and neurotransmitter, and imagine what speech and actions would result when the laws of physics operate upon that configuration of material...well, I'm not likely to get very far.

But in fact, most of us can predict with some accuracy what Barack Obama will do. He will do the sorts of things that get him re-elected, the sorts of things which increase the prestige of the Democratic Party relative to the Republican Party, the sorts of things that support American interests relative to foreign interests, and the sorts of things that promote his own personal ideals. He will also satisfy some basic human drives like eating good food, spending time with his family, and sleeping at night. If someone asked us whether Barack Obama will nuke Toronto tomorrow, we could confidently predict he will not, not because we know anything about Obama's source code, but because we know that nuking Toronto would be counterproductive to his goals.

What applies to Obama applies to all other humans. We rightly despair of modeling humans as behavior-executors, so we model them as utility-maximizers instead. This allows us to predict their moves and interact with them fruitfully. And the same is true of other agents we model as goal-directed, like evolution and the heart. It is beyond the scope of most people (and most doctors!) to remember every single one of the reflexes that control heart output and how they work. But because evolution designed the heart as a pump for blood, if you assume that the heart will mostly do the sort of thing that allows it to pump blood more effectively, you will rarely go too far wrong. Evolution is a more interesting case - we frequently model it as optimizing a species' fitness, and then get confused when this fails to accurately model the outcome of the processes that drive it.

Because it is so easy to model agents as utility-maximizers, and so hard to model them as behavior-executors, it is easy to make the mistake mentioned in The Blue-Minimizing Robot: to make false predictions about a behavior-executing agent by modeling it as a utility-maximizing agent.

So far, so common-sensical. Tomorrow's post will discuss whether we use the same deliberate simplification we apply to AIs, Barack Obama, evolution and the heart to model ourselves as well.

If so, we should expect to make the same mistake that the blue-minimizing robot made. Our actions are those of behavior-executors, but we expect ourselves to be utility-maximizers. When we fail to maximize our perceived utility, we become confused, just as the blue-minimizing robot became confused when it wouldn't shoot a hologram projector that was interfering with its perceived "goals".

Comment author: Alicorn 15 July 2011 08:54:42AM *  7 points [-]

Cisspecies representatives of nonhuman creatures have not stepped forward to speak about what their internal experience is like such that humans claiming to really be those species on the inside may be shown not to be making shit up.

If they did, it would be an empirical question whether your human-shaped Klingon-minded person reported experiences relevantly similar to regular Klingons.

Comment author: Yvain 15 July 2011 10:39:46AM 12 points [-]

If transgender people were found to report experiences and emotions significantly different from those of people born in the gender they were trans-ing to, would that convince you that transgender people are "making shit up" and so do not deserve to be taken seriously?

In response to comment by Yvain on My true rejection
Comment author: arundelo 15 July 2011 02:28:39AM *  7 points [-]

Moldbug's Fhnargl thought experiment

Fnargl.

In response to comment by arundelo on My true rejection
Comment author: Yvain 15 July 2011 10:36:02AM 7 points [-]

[Yvain crosses "get corrected on spelling of 'Fnargl'" off his List Of Things To Do In Life]

In response to comment by Yvain on My true rejection
Comment author: dripgrind 15 July 2011 12:56:25AM 1 point [-]

To build a superintelligence that actually maximizes IBM's share price in a normal way that the CEO of IBM would >approve of would require solving the friendly AI problem but then changing a couple of lines of code.

That assumes that being Friendly to all of humanity is just as easy as being Friendly to a small subset.

Surely it's much harder to make all of humanity happy than to make IBM's stockholders happy? I mean, a FAI that does the latter is far less constrained, but it's still not going to convert the universe into computronium.

Comment author: Yvain 15 July 2011 01:36:34AM 5 points [-]

Not really. "Maximize the utility of this one guy" isn't much easier than "Maximize the utility of all humanity" when the real problem is defining "maximize utility" in a stable way. If it were, you could create a decent (though probably not recommended) approximation to Friendly AI problem just by saying "Maximize the utility of this one guy here who's clearly very nice and wants what's best for humanity."

There are some serious problems with getting something that takes interpersonal conflicts into account in a reasonable way, but that's not where the majority of the problem lies.

I'd even go so far as to say that if someone built a successful IBM-CEO-utility-maximizer it'd be a net win for humanity, compared to our current prospects. With absolute power there's not a lot of incentive to be an especially malevolent dictator (see Moldbug's Fhnargl thought experiment for something similar) and in a post-scarcity world there'd be more than enough for everyone including IBM executives. It'd be sub-optimal, but compared to Unfriendly AI? Piece of cake.

In response to My true rejection
Comment author: Yvain 15 July 2011 12:04:52AM 17 points [-]

If IBM makes a superintelligent AI that wants to maximize their share price, it will probably do something less like invent brilliant IBM products, and more like hack the stock exchange, tell its computers to generate IBM's price by calling on a number in the AI's own memory, and then convert the universe to computronium in order to be able to represent as high a number as possible.

To build a superintelligence that actually maximizes IBM's share price in a normal way that the CEO of IBM would approve of would require solving the friendly AI problem but then changing a couple of lines of code. Part of what SIAI should be (and as far as I know, is) doing is trying to convince people like selfish IBM researchers that making an UnFriendly superintelligence would be a really bad idea even by their own selfish standards.

Another part is coming up with some friendly AI design ideas so that, if IBM is unusually sane and politicians are unusually sane and everyone is sane and we can make it to 2100 without killing ourselves via UnFriendly AI, then maybe someone will have a Friendly AI in the pipeline so we don't have to gamble on making it to 2200.

Also, the first rule of SIAI's assassinate unfriendly AI researchers program is don't talk about the assassinate unfriendly AI researchers program.

Comment author: Wei_Dai 14 July 2011 08:25:57PM *  5 points [-]

I don't think the purpose of the ego syntonic/dystonic distinction is to get rid of guilt, but to create a more socially acceptable framing, which may accompany guilt.

I'm confused because in the post you wrote:

Under Trivers' hypothesis, the compulsion for heroin works the same way. The heroin addict's definitely going to get that heroin, but by presenting the desire in the form of an external compulsion, the unconscious saves the heroin addict from the guilt of "choosing" heroin.

Which makes it sound like the purpose of the ego syntonic/dystonic distinction is to get rid of guilt. Are you actually saying that the purpose is really to create a more socially acceptable framing, but as a side effect it might sometimes alleviate guilt?

Comment author: Yvain 14 July 2011 09:03:25PM *  6 points [-]

I think this is a case where I knew what I meant and so I wrote something that activated the appropriate ideas in my head but was misleading and unable to communicate the appropriate ideas. I've fixed it by changing "guilt" to "social stigma".

Comment author: [deleted] 14 July 2011 12:15:36AM 17 points [-]

Ok, but then what?

I mean, this idea sounds about right to me (in general terms) and has direct consequences for my life. I'm in such a conflict right now, so I'm very motivated to find a way to deal with it.

(In case that you're addressing this in an upcoming post soon, feel free to ignore this comment until then.)

I'm currently facing opposition between two different personalities and their associated goals. Roughly speaking, one side has a very stable, mildly entertained, completely withdrawn lifestyle that takes over the whole day, and the other has fairly specific, time-intensive, but exciting ambitions and is seething with hatred over all the wasted potential and seeming incompetence and tyranny of the dominant slacker side. (I'm willing to be arbitrarily explicit, but don't want to derail into my crap unasked.) The situation is quite similar to the heroin addict, though not yet as unstable. Either side makes it look to me and to others as if it were the victim, exactly as you describe.

The conflict is clearly causing me much suffering and is not a satisfactory state of affairs, but it's not clear at all to me what I can do about it. Here's a (probably incomplete) list of things I've unsuccessfully tried over the last years (please note that I'm compressing quite some time and am focusing directly on the conflict, so this sounds probably more intense than it really is):

  1. Accept fully the desires of both sides as valid. They are not, strictly speaking, in conflict. I can totally imagine spending, say, working some hours on a long-term goal and some other hours relaxing, or maybe find ways to integrate them. This does nothing. Sure, I don't hate myself for the conflict anymore, but it doesn't resolve it and one side is still utterly dominant for months until the other violently takes over.

  2. Try to trace exactly what the conflict is. This was specifically influenced by the Method of Levels and PCT, according to which there must be 2 (or more) control systems that attempt to control for contradictory reference values. I've spent plenty of hours on that, but can't actually identify any direct conflict. I do not get a situation where I can feel both sides pulling in different directions. Rather, I get completely mind-wiped by one side, which takes over, conflict-free, then background suffering builds up, another mind-wipe, the other side rules. If I try to become aware of the background suffering, I just get a completely unfocused, unspecific build-up of pressure in my head and stomach that is intensely painful. I can concentrate on this for hours, but get no further insight or connection to anything whatsoever out of it. (I have found MoL-like analysis useful for other minor issues, though.)

  3. Different kinds of meditation. Vipassana. Concentration. Noting. Chanting. Prayer. Trying to be aware of as many internal events as possible. Ditto external. Focusing on rest. All this leads either nowhere or gets the results described in 1) and 2).

  4. Drugs. Anti-depressants, uppers, downers, psychedelics, dissociatives, omega 3, whatever. Some make it feel like I'm resolving something (particularly the entheogens), but nothing changes. I always walk away with "oh, that's why I'm suffering, now I got it!" and a few days later, at most, it's all back.

  5. Contracts. I've set up multiple contracts with myself, trying to negotiate conditions that are getting closer to resolution, but they hold no power and after a few days I start abandoning them, regardless of how severe the contract was.

  6. Finding out what "my purpose" or "my real values" are, by introspection and writing. I don't even get fake results that seem convincing at first, like with the drugs. I have lots and lots of pages of psycho-analytical babble, though.

  7. Using CBT and The Work to repair broken or harmful thoughts. Did absolutely nothing for me. Most of the time, I don't even have thoughts. (This wasn't always the case. When I first meditated at 15, I found it hard to sit for 5 minutes and not get overwhelmed by my thoughts. Now I go blank quite often and then get bored.) And the thoughts I found were mostly not about me (but about stuff I read, like what's good about Python, or explaining to myself how spaced repetition works so that I don't forget the explanation). The few worries I could dig out were not really responsive to the approach. If I try investigating my feelings using these techniques, I simply get no results. "What's this anxiety about?", "What does it predict?", "Who would I be without it?", all just gets a mental shrug and "don't know / don't care". I might as well ask what an itch is "about" or "who I would be without it".

  8. Using operant conditioning, trivial inconveniences and similar "behaviorist" tools to get rid of one side or keep it in check. (I've tried this with both sides, at different times of course.) No results, except that I changed a few food preferences in the process. Essentially every single akrasia tactic described on LW fails me here.

  9. Blatantly and totally picking sides, embracing one personality as the only true one. (Again, did this with each side eventually.) This lead me to actively harm the other, often destroying things, leaving associated communities, the like. Of course, I made my decision public and took full responsibility for the previous conflict and for the new direction I was going to take. This feels great for typically a few weeks (and can trigger quite a manic episode), but then the other side comes back, "willpower" gets drained and I descend into total apathy. There isn't even active opposition anymore; both sides are perfectly willing to commit "suicide" by now and let the other side take over, as long as this damn suffering finally ends, but they don't go anywhere.

  10. Give up. I've sat down, stared at a wall and said to myself, "I'm tired of this bullshit and will do nothing anymore! I won't influence anything, won't want anything, prefer nothing, decide nothing, just sit here and stare at this wall. Whatever happens then, I don't care.". I spent a few hours more or less catatonic, then had some kind of peace for a few days, but the background suffering just builds up again and the anxiety, self-hate and pain come back, making it impossible to enjoy anything.

I hope this isn't too incoherent or tl;dr, but I'm frustrated that this looks as if there should be some deep understanding and solution that would actually help me, not just a fancy idea that signals how cynical I am.

In response to comment by [deleted] on Ego syntonic thoughts and values
Comment author: Yvain 14 July 2011 08:06:36PM 1 point [-]

This is sufficiently different from my experience that I don't think I can help you.

My experience is not of two equal and opposite "personalities", but of some consciously held beliefs and goals that I'm more or less successful at living up to. Do you identify equally with both these "personalities", or do you consciously support one of them and assume the other one exists because of your actions?

Comment author: Wei_Dai 13 July 2011 11:30:19PM *  7 points [-]

It sounds like you're saying that evolution built a self-punishment system (which we call "guilt"), but then built another mechanism that let's certain behaviors escape that self-punishment system (by having our minds view them as compulsions). Is that right?

Can you explain why it might be evolutionarily advantageous to visibly self-punish some socially unacceptable behaviors, but not others? And what criteria does the brain actually use to make this (subconscious) choice?

Comment author: Yvain 14 July 2011 08:01:32PM 2 points [-]

Thanks for the rephrasing. I now understand MixedNuts' question better.

I don't think the purpose of the ego syntonic/dystonic distinction is to get rid of guilt, but to create a more socially acceptable framing, which may accompany guilt.

Consider the case of a pastor with homosexual urges. If they're mild, he can hide it from himself. If they're stronger, and he engages in homosexual behavior, he can't. If he gets caught, then he'll probably phrase it as "I get these temptations, can't do anything about it" instead of as "Yeah, turns out I'm a homosexual and following the Bible isn't that important to me", go to some sort of 'therapy', and be forgiven while continuing to hold most of his previous beliefs about himself. He'll probably still feel guilty about it either way.

Comment author: Yvain 13 July 2011 10:59:22PM *  11 points [-]

Another possibility is that people really don't like the idea of having to do a 2 to 4 hour deliberately useless test assignment.

Comment author: MixedNuts 13 July 2011 09:28:37PM 11 points [-]

By presenting the antidote to Lisa in the form of an external compulsion, he allows Lisa to do what she wanted anyway and avoid feeling guilty.

Wait, what? What on Earth is the purpose of guilt if you can avoid it? Though it does disminish guilt, so it's a mechanism that removes some guilt for actions you're going to continue despite guilt... but what for? It it completely removed guilt, might as well not generate guilt for actions reinforced that strongly. (And if not, why does it exist?) And it's not like you don't continue - so why would it replace guilt as a signal of cooperation?

One of the classic stereotypes of the insane is that they hear voices telling them to kill people.

Two hypotheses: Someone with voices saying "Kill them!" is more likely than someone with voices saying "Let's go fishing!" to be institutionalized. Mental illness is influenced by culture, which propagates this cliché.

Anecdote: Once I had a voice. It told me to do minor random stuff - cross the street at an unusual point, take a parallel street, explore a part of town. I obeyed because why not. Then out of the blue it told me to kill the next person I saw. I laughed at the obvious cliché and pushed the voice away until it was gone.

Comment author: Yvain 13 July 2011 09:42:30PM *  4 points [-]

Wait, what? What on Earth is the purpose of guilt if you can avoid it?

I'm not sure what's surprising you here. You don't think John would feel more guilty if he had to beg Lisa for an antidote that she also wanted, than if Lisa said "I gave it to you this morning in your coffee, the decision was never in your hands"?

You can find more on what I think of guilt here.

As for your voice - did it sound more like an intrusive thought? Or did it sound like an external person talking, heard exactly as loudly and clearly as if it were your mother or your friend talking to you? Was it your own voice, the voice of someone you knew, or a stranger?

Comment author: Vladimir_M 12 July 2011 11:10:07PM 9 points [-]

​I see at least two problems with this case study.

First, what sort of sampling bias is introduced by studying only men who are willing to view such materials? It seems highly implausible to me that this effect is zero.

Second, if true, this theory should generalize to other cases of people who express an exceptionally strong opposition towards some low-status/disreputable behavior that can be practiced covertly, or some low-status beliefs that can be held in secret. Yet it's hard for me to think of any analogous examples that would be the subject of either folk theories or scientific studies.

In fact, this generalization would lead to the conclusion that respectable high-status activists who crusade against various behaviors and attitudes that are nowadays considered disreputable, evil, dangerous, etc., should be suspected that they do it because they themselves engage in such behaviors (or hold such attitudes) covertly. The funny thing is, in places and social circles where homophobia is considered disreputable, this should clearly apply to campaigners against homophobia!

Comment author: Yvain 13 July 2011 08:26:20PM *  10 points [-]

Second, if true, this theory should generalize to other cases of people who express an exceptionally strong opposition towards some low-status/disreputable behavior that can be practiced covertly, or some low-status beliefs that can be held in secret. Yet it's hard for me to think of any analogous examples that would be the subject of either folk theories or scientific studies.

There are a few other scientific results of this type: search the literature under "reaction formation". For example:

Morokoff (1985): Women high in self-reported "sex guilt" have lower self-reported reaction to erotic stimuli but higher physiological arousal.

Dutton & Lake (1976): Whites with no history of prejudice and self-reported egalitarian beliefs were given bogus feedback during a task intended to convince them they were subconsciously prejudiced (falsely told that they had high skin response ratings of fear/anger when shown slides of interracial couples). After they had left the building, they were approached by either a black or white beggar. Whites who had received the false racism feedback gave more to the black beggar (though not to the white beggar) than whites who had not.

Sherman and Garkin (1980): Subjects were asked to solve a difficult riddle in which the trick answer involved sex-roles, such that after failing they felt "implicitly accused of sexism" (couldn't find the exact riddle, but I imagine something like this). Afterwards they were asked to evaluate a sex-discrimination case. People who had previously had to solve the riddle gave harsher verdicts against a man accused of sexual discrimination than those who had not.

I've heard anecdotal theories of a few similar effects - for example, that the loudest and most argumentative religious believers are the ones who secretly doubt their own faith.

Overall I probably shouldn't have included the case study because I don't think Trivers' theory stands or falls on this one point, and it's probably not much more than tangential to the whole idea of a conscious/unconscious divide.

Trivers on Self-Deception

33 Yvain 12 July 2011 09:04PM

People usually have good guesses about the origins of their behavior. If they eat, we believe them when they say it was because they were hungry; if they go to a concert, we believe them when they say they like the music, or want to go out with their friends. We usually assume people's self-reports of their motives are accurate.

Discussions of signaling usually make the opposite assumption: that our stated (and mentally accessible) reasons for actions are false. For example, a person who believes they are donating to charity to "do the right thing" might really be doing it to impress others; a person who buys an expensive watch because "you can really tell the difference in quality" might really want to conspicuously consume wealth.

Signaling theories share the behaviorist perspective that actions do not derive from thoughts, but rather that actions and thoughts are both selected behavior. In this paradigm, predicted reward might lead one to signal, but reinforcement of positive-affect producing thoughts might create the thought "I did that because I'm a nice person".

Robert Trivers is one of the founders of evolutionary psychology, responsible for ideas like reciprocal altruism and parent-offspring conflict. He also developed a theory of consciousness which provides a plausible explanation for the distinction between selected actions and selected thoughts.

TRIVERS' THEORY OF SELF-DECEPTION

Trivers starts from the same place a lot of evolutionary psychologists start from: small bands of early humans grown successful enough that food and safety were less important determinants of reproduction than social status.

The Invention of Lying may have been a very silly movie, but the core idea - that a good liar has a major advantage in a world of people unaccustomed to lies - is sound. The evolutionary invention of lying led to an "arms race" between better and better liars and more and more sophisticated mental lie detectors.

There's some controversy over exactly how good our mental lie detectors are or can be. There are certainly cases in which it is possible to catch lies reliably: my mother can identify my lies so accurately that I can't even play minor pranks on her anymore. But there's also some evidence that there are certain people who can reliably detect lies from any source at least 80% of the time without any previous training: microexpressions expert Paul Ekman calls them (sigh...I can't believe I have to write this) Truth Wizards, and identifies them at about one in four hundred people.

The psychic unity of mankind should preclude the existence of a miraculous genetic ability like this in only one in four hundred people: if it's possible, it should have achieved fixation. Ekman believes that everyone can be trained to this level of success (and has created the relevant training materials himself) but that his "wizards" achieve it naturally; perhaps because they've had a lot of practice. One can speculate that in an ancestral environment with a limited number of people, more face-to-face interaction and more opportunities for lying, this sort of skill might be more common; for what it's worth, a disproportionate number of the "truth wizards" found in the study were Native Americans, though I can't find any information about how traditional their origins were or why that should matter.

If our ancestors were good at lie detection - either "truth wizard" good or just the good that comes from interacting with the same group of under two hundred people for one's entire life - then anyone who could beat the lie detectors would get the advantages that accrue from being the only person able to lie plausibly.

Trivers' theory is that the conscious/unconscious distinction is partly based around allowing people to craft narratives that paint them in a favorable light. The conscious mind gets some sanitized access to the output of the unconscious, and uses it along with its own self-serving bias to come up with a socially admirable story about its desires, emotions, and plans. The unconscious then goes and does whatever has the highest expected reward - which may be socially admirable, since social status is a reinforcer - but may not be.

HOMOSEXUALITY: A CASE STUDY

It's almost a truism by now that some of the people who most strongly oppose homosexuality may be gay themselves. The truism is supported by research: the Journal of Abnormal Psychology published a study measuring penile erection in 64 homophobic and nonhomophobic heterosexual men upon watching different types of pornography, and found significantly greater erection upon watching gay pornography in the homophobes. Although somehow this study has gone fifteen years without replication, it provides some support for the folk theory.

Since in many communities openly declaring one's self homosexual is low status or even dangerous, these men have an incentive to lie about their sexuality. Because their facade may not be perfect, they also have an incentive to take extra efforts to signal heterosexuality by for example attacking gay people (something which, in theory, a gay person would never do).

Although a few now-outed gays admit to having done this consciously, Trivers' theory offers a model in which this could also occur subconsciously. Homosexual urges never make it into the sanitized version of thought presented to consciousness, but the unconscious is able to deal with them. It objects to homosexuality (motivated by internal reinforcement - reduction of worry about personal orientation), and the conscious mind toes party line by believing that there's something morally wrong with gay people and only I have the courage and moral clarity to speak out against it.

This provides a possible evolutionary mechanism for what Freud described as reaction formation, the tendency to hide an impulse by exaggerating its opposite. A person wants to signal to others (and possibly to themselves) that they lack an unacceptable impulse, and so exaggerates the opposite as "proof".

SUMMARY

Trivers' theory has been summed up by calling consciousness "the public relations agency of the brain". It consists of a group of thoughts selected because they paint the thinker in a positive light, and of speech motivated in harmony with those thoughts. This ties together signaling, the many self-promotion biases that have thus far been discovered, and the increasing awareness that consciousness is more of a side office in the mind's organizational structure than it is a decision-maker.

Comment author: syllogism 10 July 2011 08:34:06AM *  10 points [-]

Deal. I'll even pull rank with my formal qualifications on English grammar, should they care about that.

Comment author: Yvain 12 July 2011 08:45:47PM *  2 points [-]

Question for the formally qualified grammarian: When using singular "they", which is correct?

  1. "When a person is biased, they make mistakes"

  2. "When a person is based, they makes mistakes"

The second sounds absolutely horrible, but if singular "they" is really being used as a singular in the same sense as "he" or "she", it sounds like it ought to be correct.

Comment author: jimmy 12 July 2011 12:07:56AM *  5 points [-]

I have several issues with the ideas you present here. Of course, it's likely that it's just another communication error given our last conversation..

Here "involuntary" needs to be distinguished from "hard-to-resist". Most people do not define smoking as an involuntary behavior, because, although people may smoke even when they wish they wouldn't, they have the feeling that they could have chosen not to smoke, they just didn't.

For smoking, sure. I have a habit of twirling my hair which can be damn annoying and sometimes hard to resist. I can choose to keep myself from doing it by paying attention to what my hands are doing and forcing it to stop. If I'm concentrating on other things, it happens without me realizing it. It seems out of touch with the normal meaning of the word "voluntary" to include this.

But when our masked gunman tells me to increase my body temperature by two degrees or he'll shoot, he is out of luck.

Well, I just tried it. I got 0.6 degrees increase. If I were to put blankets on like feverish people normally do, I'm sure I could do better. If there was a real gunman, I'd start doing squat jumps. Marathon runners can get their core up to 105.

Does that mean that the process of regulating core temp both is and is not "voluntary", depending on the size of the temperature change?

If you're not willing to do the work of trying to model the hierarchical structure of the brain and the interconnections, what can this theory say about why my locus of voluntary control is bigger than it used to be? Can your black box theory advise people on how to increase their locus of control? What can you even use it for?

But an explanation in the spirit of reinforcement learning would have to start by insisting on treating thoughts and emotions as effects rather than causes. Instead of explaining my choice of restaurant by saying I thought about it and decided McDonalds was best, it would be more accurate to say that previous experiences with McDonalds caused both the thought "I should go to McDonalds" and the behavior of going to McDonalds.

What's the evolutionary purpose of thoughts if they don't do anything? Where do thoughts like "I just thought about going to McDonalds" come from? What distinguishes this from philosophical zombies?

Comment author: Yvain 12 July 2011 04:09:04AM 3 points [-]

I know that I can't currently change my temperature by willing it as I've tried a few times, but you may just be better at it. Certainly people can train themselves to do so with eg biofeedback. So if you in fact were able to raise your temperature .6 degrees just by willing it, I'm willing to admit that temperature change is partly voluntary (partly because I'm guessing even if you wanted to you couldn't do 10 degrees).

But doing things like wearing a blanket or doing squat jumps are "cheating". Sleepwalking is also "voluntary", if you mean that if you didn't want to do it, you could tie yourself to your bed, and humans can fly if they're allowed to use airplanes. But that seems to be a case of blanket-wearing being voluntary, and temperature rising automatically in that condition. Even something like visualizing yourself in the Arctic so your temperature rises to compensate is "cheating" of a sort - we all agree that visualizing things is a voluntary behavior.

What's the evolutionary purpose of thoughts if they don't do anything? Where do thoughts like "I just thought about going to McDonalds" come from? What distinguishes this from philosophical zombies?

Tomorrow's post.

Comment author: Jonathan_Graehl 12 July 2011 12:07:22AM *  2 points [-]

actions may not end up so harmonious, because they suffer from differential reinforcement.

What does "differential reinforcement" mean? If you covered it in the previous post, a link or summary would be good.

From Wikipedia:

Other simple (reinforcement) schedules include:

Differential reinforcement of incompatible behavior – Used to reduce a frequent behavior without punishing it by reinforcing an incompatible response. An example would be reinforcing clapping to reduce nose picking.

Differential reinforcement of other behavior (DRO) – Also known as omission training procedures, an instrumental conditioning procedure in which a positive reinforcer is periodically delivered only if the participant does something other than the target response. An example would be reinforcing any hand action other than nose picking.

Differential reinforcement of low response rate (DRL) – Used to encourage low rates of responding. It is like an interval schedule, except that premature responses reset the time required between behavior.

Lab example: DRL10" = a rat is reinforced for the first response after 10 seconds, but if the rat responds earlier than 10 seconds there is no reinforcement and the rat has to wait 10 seconds from that premature response without another response before bar pressing will lead to reinforcement.

Real world example: "If you ask me for a potato chip no more than once every 10 minutes, I will give it to you. If you ask more often, I will give you none."

Differential reinforcement of high rate (DRH) – Used to increase high rates of responding. It is like an interval schedule, except that a minimum number of responses are required in the interval in order to receive reinforcement. Lab example: DRH10"/15 responses = a rat must press a bar 15 times within a 10 second increment to get reinforced Real world example: "If Lance Armstrong is going to win the Tour de France he has to pedal x number of times during the y-hour race."

Comment author: Yvain 12 July 2011 04:02:47AM 0 points [-]

Sorry, should have said "different" instead of "differential". Changed.

Voluntary Behavior, Conscious Thoughts

24 Yvain 11 July 2011 10:13PM

Skinner proposes a surprisingly easy way to dissolve the problem of what it means for an action to be "voluntary", or "under voluntary control".

We commonly perceive certain actions as under voluntary control: for example, I can control what words I'm typing right now, or whether I go out for dinner tonight. Other actions are not under voluntary control: for example, absent some exciting technique like biofeedback I can't control my heartbeat or my core body temperature or the amount of bile produced by my liver.

Other, larger-scale actions also get classified as involuntary. Many people consider sleepwalking involuntary, including the bizarre "sleep-eating" behaviors some people display on Ambien and related drugs. The tics of Tourette's are involuntary. Our emotions and preferences are at least a little involuntary: office workers might like to be able to will away their boredom, or mourners their sorrow, but most can't.

Here "involuntary" needs to be distinguished from "hard-to-resist". Most people do not define smoking as an involuntary behavior, because, although people may smoke even when they wish they wouldn't, they have the feeling that they could have chosen not to smoke, they just didn't.

The philosophy of voluntary versus involuntary behavior seems to run up against a wall when it hits the question of "what is truly me?". If we make the reductionist identification of "me" with "my brain", well, clearly it's my brain controlling sleepwalking and boredom, but it still doesn't feel like I am controlling these things. Trying to go deeper ends up hopelessly vague, usually with talk of "higher level brain processes" versus "lower level brain processes" and an identification of "myself" with the higher ones. There may be a role for this kind of talk, but it couldn't hurt to look for something more explanatory.

Skinner, true to his quest, explains the distinction without any discussion of "brain processes" or "self". He says that voluntary behavior is behavior subject to operant conditioning, and involuntary behavior is everything else.

It might be clearer to define voluntary behavior as fully transparent to reinforcement. Imagine a man with a gun, threatening to shoot me if I go out for dinner tonight. The fear of punishment will be effective: I'll avoid going out. Lust for reward, too, would be effective. If Bill Gates offered me $1 billion to stay in, that's what I'd do.

But when our masked gunman tells me to increase my body temperature by two degrees or he'll shoot, he is out of luck. And no matter how much money Bill Gates offers me for same, he can't make me give myself a fever either.

There is a place, too, for the hard-to-resist behaviors in all this: these are behaviors which can be affected by reward, but as yet have not been. If a masked man held his gun to the head of smokers and told them to stop or he'd shoot, they would stop. But thus far, none of the potential rewards of not smoking have been sufficient to change smokers' behavior.

CONSCIOUSNESS

The idea of voluntary behavior is tied so intimately to the idea of the self, or of consciousness (the easy problem, not the hard one), that one would hope that a new approach to one might be able to shed some light on the other. If voluntary action depends on transparency to reinforcement, where does that leave consciousness?

I haven't been able to find Skinner's beliefs on this subject (when he talks about consciousness, it's usually to deny it as an ontologically fundamental entity) and I've never seen anywhere near as elegant a reduction. But an explanation in the spirit of reinforcement learning would have to start by insisting on treating thoughts and emotions as effects rather than causes. Instead of explaining my choice of restaurant by saying I thought about it and decided McDonalds was best, it would be more accurate to say that previous experiences with McDonalds caused both the thought "I should go to McDonalds" and the behavior of going to McDonalds.

There is an intuitive connection between thought and language, and Soviet psychologist Lev Vygotsky made the connection more explicit; he found that children begin by speaking their stream of consciousness aloud to inform other people, and eventually learn to suppress that stream into nonvocal (subvocal?) thought.

The last post in this sequence discussed different reinforcement of thought and action. Speech and thought make a natural category as opposed to action; both are fast and easy, and so less likely to be affected by time and effort discounting. Both are point actions as opposed to a long project like learning Swahili or quitting smoking. And both bring reinforcement not through normal sensory channels (saying a word doesn't give pleasure in the same way smoking a cigarette might, nor pain in the same way having to study a boring grammar textbook might) but in what they say about you as a person and how they affect other people's real (and perceived) opinion of you.

So even if there is no governor anywhere unifying all thoughts and words, they may come out in harmony because they were selected by the same processes for the same reasons. And actions may not end up so harmonious, because they suffer from differential reinforcement.

Such harmony resembles the idea of a core "me", of whom all my thoughts are a part, and who has complete power over my organs of speech - but who is sometimes at odds with my actions or emotions.

The reinforcement governing thought and speech is most likely to be internal reinforcement based on your own self-perception and on others' perception of you. If there's a good reason reputation management processes need to be different from decision-making processes, understanding that difference could help understand the evolutionary history of a perceived difference between the conscious and unconscious mind. One such reason is provided by Robert Trivers' theory of social consciousness, the subject of tomorrow's post.

Physical and Mental Behavior

48 Yvain 10 July 2011 08:20PM

B.F. Skinner called thoughts "mental behavior". He believed they could be rewarded and punished just like physical behavior, and that they increased or declined in frequency accordingly.

Sadly, psychology has not yet advanced to the point where we can give people electric shocks for thinking things, so the sort of rewards and punishments that reinforce thoughts must be purely internal reinforcement. A thought or intention that causes good feelings gets reinforced and prospers; one that causes bad feelings gets punished and dies out.

(Roko has already discussed this in Ugh Fields; so much as thinking about an unpleasant task is unpleasant; therefore most people do not think about unpleasant tasks and end up delaying them or avoiding them completely. If you haven't already read that post, it does a very good job of making reinforcement of thoughts make sense.)

A while back, D_Malik published a great big List Of Things One Could Do To Become Awesome.  As David_Gerard replied, the list was itself a small feat of awesome. I expect a couple of people started on some of the more awesome-sounding entries, then gave up after a few minutes and never thought about it again. Why?

When I was younger, I used to come up with plans to become awesome in some unlikely way. Maybe I'd hear someone speaking Swahili, and I would think "I should learn Swahili," and then I would segue into daydreams of being with a group of friends, and someone would ask if any of us spoke any foreign languages, and I would say I was fluent in Swahili, and they would all react with shock and tell me I must be lying, and then a Kenyan person would wander by, and I'd have a conversation with them in Swahili, and they'd say that I was the first American they'd ever met who was really fluent in Swahili, and then all my friends would be awed and decide I was the best person ever, and...

...and the point is that the thought of learning Swahili is pleasant, in the same easy-to-visualize but useless way that an extra bedroom for Grandma is pleasant. And the intention to learn Swahili is also pleasant, because it will lead to all those pleasant things.  And so, by reinforcement of mental behavior, I continue thinking about and intending to learn Swahili.

Now consider the behavior of studying Swahili. I've never done so, but I imagine it involves a lot of long nights hunched over books of Swahili grammar. Since I am not one of the lucky people who enjoys learning languages for their own sake, this will be an unpleasant task. And rewards will be few and far between: outside my fantasies, my friends don't just get together and ask what languages we know while random Kenyans are walking by.

In fact, it's even worse than this, because I don't exactly make the decision to study Swahili in aggregate, but only in the form of whether to study Swahili each time I get the chance. If I have the opportunity to study Swahili for an hour, this provides no clear reward - an hour's studying or not isn't going to make much difference to whether I can impress my friends by chatting with a Kenyan - but it will still be unpleasant to spend an hour of going over boring Swahili grammar. And time discounting makes me value my hour today much more than I value some hypothetical opportunity to impress people months down the line; Ainslie shows quite clearly I will always be better off postponing my study until later.

So the behavior of actually learning Swahili is thankless and unpleasant and very likely doesn't happen at all.

Thinking about studying Swahili is positively reinforced, actually studying Swahili is negatively reinforced. The natural and obvious result is that I intend to study Swahili, but don't.

The problem is that for some reason, some crazy people expect for the reinforcement of thoughts to correspond to the reinforcement of the object of those thoughts. Maybe it's that old idea of "preference": I have a preference for studying Swahili, so I should satisfy that preference, right? But there's nothing in my brain automatically connecting this node over here called "intend to study Swahili" to this node over here called "study Swahili"; any association between them has to be learned the hard way.

We can describe this hard way in terms of reinforcement learning: after intending to learn Swahili but not doing so, I feel stupid. This unpleasant feeling propagates back to its cause, the behavior of intending to learn Swahili, and negatively reinforces it. Later, when I start thinking it might be neat to learn Mongolian on a whim, this generalizes to behavior that has previously been negatively reinforced, so I avoid it (in anthropomorphic terms, I "expect" to fail at learning Mongolian and to feel stupid later, so I avoid doing so).

I didn't learn this the first time, and I doubt most other people do either. And it's a tough problem to call, because if you overdo the negative reinforcement, then you never try to do anything difficult ever again.

In any case, the lesson is that thoughts and intentions get reinforced separately from actions, and although you can eventually learn to connect intentions to actions, you should never take the connection for granted.

Comment author: syllogism 10 July 2011 06:37:09AM 25 points [-]

I've been meaning to make a post about this small procedural note. Singular they has a long history in English as a gender-neutral third person singular pronoun. Languages tend to resist the introduction of new pronouns, as they're "closed class" --- part of the language's grammar. It's especially problematic that nobody can even agree on which invented pronoun to get behind!

Can't we all just use singular they? It's much nicer.

Comment author: Yvain 10 July 2011 06:58:44AM *  31 points [-]

Okay, okay, I'll use singular they if you all promise that the first time someone pompously chides me for using "they" in the singular, you'll give them at least as much trouble as you're giving me for using gender-neutral third person pronouns.

Comment author: Jonathan_Graehl 09 July 2011 12:02:20AM 11 points [-]

I agree that known biases can be explained by curves like those, plus the choice of a "frame". But how do we know we're not overfitting?

In other words: does prospect theory pay rent?

I'd want to at least see that we're identifying some real differences between people when we fit their curves from a bunch of measurements of their behavior - I'd expect their personally fit model to describe their (held-out from fitting) future actions better than one fit over the whole population, etc.

It seems like the additional degree of freedom "well, it depends on how they chose their frame in this instance" needs to be nailed down as part of testing the model's fit on future actions.

Comment author: Yvain 10 July 2011 06:22:18AM *  7 points [-]

I am not entirely qualified to answer this objection, and I hope that one day someone who is more mathematical will make a post on the exact math involved.

Until then, I would say that the important part of prospect theory is not fitting numbers to the curves or determining the exact curve for each different person, but the discovery that the curves have the same basic shape in everyone. For example, that the slope of the losses curve is always greater than the slope of the gains curve; that the slope of both curves is steepest near zero but eventually levels out; that gains are always concave and losses are always convex. That subjective probability is steepest near zero, and also steep near one, but flatter in the middle. That decisions depend on frames, which can be changed and scaled depending on presentation.

I'm describing these visually because that's how I think; in the paper I linked to on top, Kahneman and Tversky describe the same information in the terms of mathematical equations which expected utility follows. None of these are intuitively predictable without having done the experiment, and all of them are pretty constant across different decisions.

I'm not sure what the status of research on applied prospect theory - figuring out the exact equations you can plug a frame and an amount of money into and predict the decision - is, but it must have had some success to win a Nobel Prize.

Prospect Theory: A Framework for Understanding Cognitive Biases

66 Post author: Yvain 10 July 2011 05:20AM

Related to: Shane Legg on Prospect Theory and Computational Finance

This post is on prospect theory partly because it fits the theme of replacing simple utility functions with complicated reward functions, but mostly because somehow Less Wrong doesn't have any posts on prospect theory yet and that needs to change.

Kahneman and Tversky, the first researchers to identify and rigorously study cognitive biases, proved that a simple version of expected utility theory did not accurately describe human behavior. Their response was to develop prospect theory, a model of how people really make decisions. Although the math is less elegant than that of expected utility, and the shapes of the curves have to be experimentally derived, it is worth a look because it successfully predicts many of the standard biases.

(source: Wikipedia)

A prospect theory agent tasked with a decision first sets it within a frame with a convenient zero point, allowing em to classify the results of the decision as either losses or gains. Ey then computes a subjective expected utility, where the subjective expected utility equals the subjective value times the subjective probability. The subjective value is calculated from the real value using a value function similar to the one on the left-hand graph, and the subjective probability is calculated from the real probability using a weighting function similar to the one on the right-hand graph.

Clear as mud? Let's fill some numbers into the functions - the exact assignments don't really matter as long as we capture the spirit of where things change steeply versus slowly - and run through an example.



Imagine a prospect theory agent - let's call him Prospero - trying to decide whether or not to buy an hurricane insurance policy costing $5000/year. Prospero owns assets worth $10,000, and estimates a 50%/year chance of a hurricane destroying his assets; to make things simple, he will be moving in one year and so need not consider the future. Under expected utility theory, he should feel neutral about the policy.

Under prospect theory, he first sets a frame in which to consider the decision; his current state is a natural frame, so we'll go with that.

We see on the left-hand graph that an objective $10,000 loss feels like a $5,000 loss, and an objective $5000 loss feels like a $4000 loss. And we see on the right-hand graph that a 50% probability feels like a 40% probability.

Now Prospero's choice is a certain $4000 loss if he buys the insurance, versus a 40% chance of a $5000 loss if he doesn't. Buying has a subjective expected utility of -$4000; not buying has a subjective expected utility of -$2000. So Prospero decisively rejects the insurance.

But suppose Prospero is fatalistic; he views his assets as already having been blown away. Here he might choose a different frame: the frame in which he starts with zero assets, and anything beyond that is viewed as a gain.

Since the gain half of the value function levels off more quickly than the loss half, $5000 is now subjectively worth $3000, and $10000 is now subjectively worth $3500.

Here he must choose between a certain gain of $5000 and a 50% chance of gaining $10000. Expected utility gives the same result as before, obviously. In prospect theory, he chooses between a certain subjective gain of $3000 and a 40% chance of gaining $3500. The insurance gives him subjective expected utility of $3000, and rejecting it gives him subjective expected utility of $1400.

All of a sudden Prospero wants the insurance.

We notice the opposite effect if there is only a a 1% chance of a hurricane. The insurance salesman lowers his price to $100 to preserve the neutrality of the insurance option when using utility.

But subjective probability rises very quickly, so a 1% chance may correspond to a subjective 10% chance. Now in the first frame, Prospero must decide between an objective loss of -$100 with certainty (corresponding to -$300 subjective since the value function is steeper closer to zero) or an objective loss of -$10,000 with objective probability 1% (subjective of 10%). Now the expected subjective utilities are -$300 if he buys, versus -$500 if he rejects. And so he buys the insurance. When we change the risk of hurricane from 50% to 1%, then even though we reduce the price of the insurance by an exactly equal amount, Prospero's decision switches from not buying to buying.

Let's see how many previously discussed biases we can fit into this model.

Prospero's change from rejecting the insurance when framed as gains, to buying it when framed as losses, directly mirrors the change in preferred survival strategies mentioned in Circular Altruism.

The necessity of frame-shifting between different perceptions of losses also produces the Sunk Cost Fallacy.

The greater steepness of the value function with losses as opposed to gains is not even an explanation for, but merely a mathematical representation of, loss aversion.

The leveling off of the value function that turned the huge objective difference between +$5000 and +$10000 into the teensy little subjective difference between +$3000 and +$3500 mirrors the scope insensitivity under which people show about the same level of interest in proposals to save endangered birds whether a thousand, ten thousand, or a hundred thousand birds are involved.

It may not be an official bias, but the "but there's still a chance, right" outlook looks a lot like the sharply rising curve of the subjective probability function near zero.

And although it is not immediately obvious from the theory, some people want to link the idea of a frame to priming and anchoring-adjustment, on the grounds that when a suitable reference frame doesn't exist any primed stimulus can help establish one.

And now, the twist: prospect theory probably isn't exactly true. Although it holds up well in experiments where subjects are asked to make hypothetical choices, it may fare less well in the rare experiments where researchers can afford to offer subjects choices for real money (this isn't the best paper out there, but it's one I could find freely available).

Nevertheless, prospect theory seems fundamentally closer to the mark than simple expected utility theory, and if any model is ever created that can explain both hypothetical and real choices, I would be very surprised if at least part of it did not involve something looking a lot like Kahneman and Tversky's model.

Comment author: timtyler 08 July 2011 10:02:50PM *  0 points [-]

It seems as though this idea is closely related to "diminishing marginal utility":

The law of diminishing marginal utility is at the heart of the explanation of numerous economic phenomena, including time preference and the value of goods.

Comment author: Yvain 09 July 2011 09:00:59PM 8 points [-]

I don't think so.

Diminishing marginal utility is a fundamentally rational process: I really do need my first $20,000 more than I need the next $100,000, because when spending the first $20,000 to increase my utility, I can knock off my low-hanging fruit preferences like food, water, and housing - but when spending the next $100,000 I come to more complicated preferences like social status and comfort that aren't quite as important.

But the discounting I'm mentioning here is per item. I would be more likely to excuse a $50 cost overrun on a $200 item than on a $20 item, even if I am a millionaire and in the end $50 makes no difference to my total amount of money either way. Even if I know I'm going to buy both a $20 item and a $200 item, I'd still prefer getting the $50 surcharge attached to the $200 item, even though it doesn't affect my total expenditure. That's irrational, and so it's got to be a bias rather than an instance of diminishing marginal utility.

Wanting vs. Liking Revisited

34 Yvain 09 July 2011 08:54PM

In Are Wireheads Happy? I discussed the difference between wanting something and liking something. More recently, Luke went deeper into some of the science in his post Not for the Sake of Pleasure Alone.

In the comments of the original post, cousin_it asked a good question: why implement a mind with two forms of motivation? What, exactly, are "wanting" and "liking" in mind design terms?

Tim Tyler and Furcas both gave interesting responses, but I think the problem has a clear answer in a reinforcement learning perspective (warning: formal research on the subject does not take this view and sticks to the "two different systems of different evolutionary design" theory). "Liking" is how positive reinforcement feels from the inside; "wanting" is how the motivation to do something feels from the inside. Things that are positively reinforced generally motivate you to do more of them, so liking and wanting often co-occur. With more knowledge of reinforcement, we can begin to explore why they might differ.

CONTEXT OF REINFORCEMENT

Reinforcement learning doesn't just connect single stimuli to responses. It connects stimuli in a context to responses. Munching popcorn at a movie might be pleasant; munching popcorn at a funeral will get you stern looks at best.

In fact, lots of people eat popcorn at a movie theater and almost nowhere else. Imagine them, walking into that movie theater and thinking "You know, I should have some popcorn now", maybe even having a strong desire for popcorn that overrides the diet they're on - and yet these same people could walk into, I don't know, a used car dealership and that urge would be completely gone.

These people have probably eaten popcorn at a movie theater before and liked it. Instead of generalizing to "eat popcorn", their brain learned the lesson "eat popcorn at movie theaters". Part of this no doubt has to do with the easy availability of popcorn there, but another part probably has to do with context-dependent reinforcement.

I like pizza. When I eat pizza, and get rewarded for eating pizza, it's usually after smelling the pizza first. The smell of pizza becomes a powerful stimulus for the behavior of eating pizza, and I want pizza much more after smelling it, even though how much I like pizza remains constant. I've never had pizza at breakfast, and in fact the context of breakfast is directly competing with my normal stimuli for eating pizza; therefore, no matter how much I like pizza, I have no desire to eat pizza for breakfast. If I did have pizza for breakfast, though, I'd probably like it.

INTERMITTENT REINFORCEMENT

If an activity is intermittently reinforced; occasional rewards spread among more common neutral stimuli or even small punishments, it may be motivating but unpleasant.

Imagine a beginning golfer. He gets bogeys or double bogeys on each hole, and is constantly kicking himself, thinking that if only he'd used one club instead of the other, he might have gotten that one. After each game, he can't believe that after all his practice, he's still this bad. But every so often, he does get a par or a birdie, and thinks he's finally got the hang of things, right until he fails to repeat it on the next hole, or the hole after that.

This is a variable response schedule, Skinner's most addictive form of delivering reinforcement. The golfer may keep playing, maybe because he constantly thinks he's on the verge of figuring out how to improve his game, but he might not like it. The same is true for gamblers, who think the next pull of the slot machine might be the jackpot (and who falsely believe they can discover a secret in the game that will change their luck; they don't like sitting around losing money, but they may stick with it so that they don't leave right before they reach the point where their luck changes.

SMALL-SCALE DISCOUNT RATES

Even if we like something, we may not want to do it because it involves pain at the second or sub-second level.

Eliezer discusses the choice between reading a mediocre book and a good book:

You may read a mediocre book for an hour, instead of a good book, because if you first spent a few minutes to search your library to obtain a better book, that would be an immediate cost - not that searching your library is all that unpleasant, but you'd have to pay an immediate activation cost to do that instead of taking the path of least resistance and grabbing the first thing in front of you.  It's a hyperbolically discounted tradeoff that you make without realizing it, because the cost you're refusing to pay isn't commensurate enough with the payoff you're forgoing to be salient as an explicit tradeoff.

In this case, you like the good book, but you want to keep reading the mediocre book. If it's cheating to start our hypothetical subject off reading the mediocre book, consider the difference between a book of one-liner jokes and a really great novel. The book of one-liners you can open to a random page and start being immediately amused (reinforced). The great novel you've got to pick up, get into, develop sympathies for the characters, figure out what the heck lomillialor or a Tiste Andii is, and then a few pages in you're thinking "This is a pretty good book". The fear of those few pages could make you realize you'll like the novel, but still want to read the joke book. And since hyperbolic discounting overcounts reward or punishment in the next few seconds, it may seem like a net punishment to make the change.

SUMMARY

This deals yet another blow to the concept of me having "preferences". How much do I want popcorn? That depends very much on whether I'm at a movie theater or a used car dealership. If I browse Reddit for half an hour because it would be too much work to spend ten seconds traveling to the living room to pick up the book I'm really enjoying, do I "prefer" browsing to reading? Which has higher utility? If I hate every second I'm at the slot machines, but I keep at them anyway so I don't miss the jackpot, am I a gambling addict, or just a person who enjoys winning jackpots and is willing to do what it takes?

In cases like these, the language of preference and utility is not very useful. My anticipation of reward is constraining my behavior, and different factors are promoting different behaviors in an unstable way, but trying to extract "preferences" from the situation is trying to oversimplify a complex situation.

Comment author: timtyler 08 July 2011 09:06:51PM *  2 points [-]

If the idea of temporal discounting can usefully be extended to effort, is there anything else it can be extended to? How about financial expenditure, for instance? Is the first penny is the hardest one?

Comment author: Yvain 08 July 2011 09:16:46PM 10 points [-]

Some friends are in the process of buying a house costing about a million dollars. There was some serious haggling over the final price, to which my friend finally replied "Forget about it, it's just thirty thousand dollars, it's not worth the conflict." And after all, paying $1,100,000 vs. $1,130,000 doesn't seem like an interesting difference.

I imagine that if they were haggling over a car that cost $20,000, they would move heaven and earth to avoid paying $30,000 more; $20,000 vs. $50,000 seems a major difference.

This seems a lot like hyperbolic discounting, where having to wait ten minutes makes a big difference if it's ten minutes from now, but very little difference if it's a year vs. a year + ten minutes. Spending $30,000 makes a big difference if it's the first $30,000, but very little if it's $1.1 million + $30,000.

See today's post on prospect theory for more.

Comment author: timtyler 08 July 2011 08:09:32AM *  3 points [-]

It makes more sense to consider time and effort discounting as describing reward functions and not utility functions.

The description of how a reward function works sounds like how a utility function works.

What distinction are you attempting to draw between a reward function and a utility function?

Comment author: Yvain 08 July 2011 09:00:48PM 3 points [-]

I think Matt Newport's comment answers it pretty well. The distinction I'm attempting to draw is between situation dependence and situation independence. If you assign 100 utils to a new car, then that car is worth 100 utils, period. You can still kinda get a utility function by saying I assign 100 utils to a new car now, but 50 utils to a new car next week. But when you assign different amounts to gaining the car vs. losing the car (loss aversion), or different amounts if you consider each facet of having the car separately than you get when you add all of them together (subadditive utility), or that you'd value winning the car in a contest a different amount than inheriting the car from a relative, then eventually it starts to look less like you're calculating a term for "value of car" and using it in different decisions, and more like you've got a number of decision-making processes, all of which return different results for different decisions involving cars.

I admit it's not a hard-and-fast distinction so much as a difference in connotations.

Comment author: rwallace 08 July 2011 06:47:26AM 5 points [-]

Apart from the uncertainty about whether an agency will deliver a future reward at all, there is also the expenditure of resources in keeping track of and following up on a debt. If you have to keep track of a debt for a year, and take action to claim it at the end of that time, that could very well cost more than $50 worth of time and energy, particularly when you take into account the need to spend resources retaliating if the debtor fails to pay. I think it is much more plausible that people use ad hoc heuristics developed to deal with these issues, than that we actually use a hyperbolic discount function.

Comment author: Yvain 08 July 2011 06:51:05AM 6 points [-]

Pigeons also discount hyperbolically.

Comment author: mattnewport 08 July 2011 02:01:45AM 8 points [-]

Do any of the studies on hyperbolic discounting attempt to show that it is not just a consequence of combining uncertainty with something like a standard exponential discounting function? That's always seemed the most plausible explanation of hyperbolic discounting to me and it meshes with what seems to be going on when I introspect on these kinds of choices.

Most of the discussions of hyperbolic discounting I see don't even consider how increasing uncertainty for more distant rewards should factor into preferences. Ignoring uncertainty seems like it would be a sub-optimal strategy for agents making decisions in the real world.

Comment author: Yvain 08 July 2011 05:17:12AM 11 points [-]

I think exponential discounting already assumes uncertainty. You need uncertainty to discount at all - if things are going to stay the same, might as well wait until later. And it doesn't intuitively lead to hyperbolic discounts - if there's a 1% chance you'll die each week, then waiting from now until next week should make you discount the same amount as waiting from ten weeks from now until eleven.

But there is a way to use uncertainty to get from exponential to hyperbolic discounting. You get exponential if you're worried about yourself dying/being unable to use the reward/etc. But if you add in the chance of the reward going away, based on a prior where you don't know anything about how likely that is, you might get hyperbolic discounting. If you don't eat an animal you've killed in the next ten minutes, then it might get stolen by hyenas. But common-sensibly, if it goes a year without being stolen by hyenas or going bad or anything, there's not much chance of hyenas suddenly coming along in the next ten minutes after that.

Time and Effort Discounting

38 Yvain 07 July 2011 11:48PM

Related to: Akrasia, hyperbolic discounting, and picoeconomics

If you're tired of studies where you inevitably get deceived, electric shocked, or tricked into developing a sexual attraction to penny jars, you might want to sign up for Brian Wansink's next experiment. He provided secretaries with a month of unlimited free candy at their workplace. The only catch was that half of them got the candy in a bowl on their desk, and half got it in a bowl six feet away. The deskers ate five candies/day more than the six-footers, which the scientists calculated would correspond to a weight gain of over 10 pounds more per year1.

Beware trivial inconveniences (or, in this case, if you don't want to gain weight, beware the lack of them!) Small modifications to the difficulty of obtaining a reward can make big differences in whether the corresponding behavior gets executed.

TIME DISCOUNTING


The best studied example of this is time discounting. When offered two choices, where A will lead to a small reward now and B will lead to a big reward later, people will sometimes choose smaller-sooner rather than larger-later depending on the length of the delay and the size of the difference. For example, in one study, people preferred $250 today to $300 in a year; it took a promise of at least $350 to convince them to wait.

Time discounting was later found to be "hyperbolic", meaning that the discount amount between two fixed points decreases the further you move those two points into the future. For example, you might prefer $80 today to $100 one week from now, but it's unlikely you would prefer $80 in one hundred weeks to $100 in one hundred one weeks. Yet this is offering essentially the same choice: wait an extra week for an extra $20. So it's not enough to say that the discount rate is a constant 20% per week - the discount rate changes depending on what interval of time we're talking about. If you graph experimentally obtained human discount rates on a curve, they form a hyperbola.

Hyperbolic discounting creates the unpleasant experience of "preference reversals", in which people can suddenly change their mind on a preference as they move along the hyperbola. For example, if I ask you today whether you would prefer $250 in 2019 or $300 in 2020 (a choice between small reward in 8 years or large reward in 9), you might say the $300 in 2020; if I ask you in 2019 (when it's a choice between small reward now and large reward in 1 year), you might say no, give me the $250 now. In summary, people prefer larger-later rewards most of the time EXCEPT for a brief period right before they can get the smaller-sooner reward.



George Ainslie ties this to akrasia and addiction: call the enjoyment of a cigarette in five minutes the smaller-sooner reward, and the enjoyment of not having cancer in thirty years the larger-later reward. You'll prefer to abstain right up until the point where there's a cigarette in front of you and you think "I should smoke this", at which point you will do so.

Discounting can happen on any scale from seconds to decades, and it has previously been mentioned that the second or sub-second level may have disproportionate effects on our actions. Eliezer concentrated on the difficult of changing tasks, but I would add that any task which allows continuous delivery of small amounts of reinforcement with near zero delay can become incredibly addictive even if it isn't all that fun (this is why I usually read all the way through online joke lists, or stay on Reddit for hours). This is also why the XKCD solution to internet addiction - an extension that makes you wait 30 seconds before loading addictive sites - is so useful.

EFFORT DISCOUNTING


Effort discounting is time discounting's lesser-known cousin. It's not obvious that it's an independent entity; it's hard to disentangle from time discounting (most efforts usually take time) and from garden-variety balancing benefits against costs (most efforts are also slightly costly). There have really been only one or two good studies on it and they don't do much more than say it probably exists and has its own signal in the nucleus accumbens.

Nevertheless, I expect that effort discounting, like time discounting, will be found to be hyperbolic. Many of these trivial inconveniences involve not just time but effort: the secretaries had to actually stand up and walk six feet to get the candy. If a tiny amount of effort held the same power as a tiny amount of time, it would go even further toward explaining garden-variety procrastination.

TIME/EFFORT DISCOUNTING AND UTILITY

Hyperbolic discounting stretches our intuitive notion of "preference" to the breaking point.

Traditionally, discount rates are viewed as just another preference: not only do I prefer to have money, but I prefer to have it now. But hyperbolic discounting shows that we have no single discount rate: instead, we have different preferences for discount rates at different future times.

It gets worse. Time discount rates seem to be different for losses and gains, and different for large amounts vs. small amounts (I gave the example of $250 now being worth $350 in a year, but the same study found that $3000 now is only worth $4000 in a year, and $15 now is worth a whopping $60 in a year). You can even get people to exhibit negative discount rates in certain situations: offer people $10 now, $20 in a month, $30 in two months, and $40 in three months, and they'll prefer it to $40 now, $30 in a month, and so on - maybe because it's nice to think things are only going to get better?

Are there utility functions that can account for this sort of behavior? Of course: you can do a lot of things just by adding enough terms to an equation. But what is the "preference" that the math is describing? When I say I like having money, that seems clear enough: preferring $20 to $15 is not a separate preference than preferring $406 to $405.

But when we discuss time discounting, most of the preferences cited are specific: that I would prefer $100 now to $150 later. Generalizing these preferences, when it's possible at all, takes several complicated equations. Do I really want to discount gains more than losses, if I've never consciously thought about it and I don't consciously endorse it? Sure, there might be such things as unconscious preferences, but saying that the unconscious just loves following these strange equations, in the same way that it loves food or sex or status, seems about as contrived as saying that our robot just really likes switching from blue-minimization to yellow-minimization every time we put a lens on its sensor.

It makes more sense to consider time and effort discounting as describing reward functions and not utility functions. The brain estimates the value of reward in neural currency using these equations (or a neural network that these equations approximate) and then people execute whatever behavior has been assigned the highest reward.


Footnotes

1: Also cited in the same Nutrition Action article: if the candy was in a clear bowl, participants ate on average two/day more than if the candy was in an opaque bowl.

Comment author: erratio 07 July 2011 06:06:35AM 1 point [-]

Damn. Oh well, noted and edited in to the original comment.

Comment author: Yvain 07 July 2011 06:11:50AM 4 points [-]

Well, like I said, all I have to go on is stuff people said on Reddit and one failed replication study I was able to find somewhere by a grad student of the guy who did the original research. The original research is certainly interesting and relevant and does speak to the problems with a very reductionist model.

This actually gets to the same problem I'm having looking up stuff on perceptual control theory, which is that I expect a controversial theory to be something where there are lots of passionate arguments on both sides, but on both PCT and Rat Park, when I've tried to look them up I get a bunch of passionate people arguing that they're great, and then a few scoffs from more mainstream people saying "That stuff? Nah." without explaining themselves. I don't know whether it's because of Evil Set-In-Their-Ways Mainstream refusing to acknowledge the new ideas, or whether they're just so completely missing the point that people think it's not worth their while to respond. It's a serious problem and I wish that "skeptics" would start addressing this kind of thing instead of debunking ghosts for the ten zillionth time.

Comment author: erratio 07 July 2011 12:22:52AM *  4 points [-]

An animal can go its whole life not wanting heroin until it's exposed to some. Then suddenly heroin becomes extraordinarily motivating and it will preferentially choose shots of heroin to food, water, or almost anything else

Rats don't always choose drugs over everything else

Summary: An experimenter thought drug addiction in rats might be linked to being kept in distressing conditions, made a Rat Park to test the idea, and found that the rats in the enriched Rat Park environment ignored the morphine on offer.

EDIT: apparently the study had methodological issues and hasn't been replicated, making the results somewhat suspect, as pointed out by Yvain below

Comment author: Yvain 07 July 2011 04:50:11AM 4 points [-]

I hate to admit I get science knowledge from Reddit, but the past few times this was posted there it was ripped apart by (people who claimed to be) professionals in the field - riddled with metholodogical errors, inconsistently replicated, et cetera. The fact that even its proponents admit the study was rejected by most journals doesn't speak well of it.

I think it's very plausible that situation contributes to addiction; we know that people in terrible situations have higher discount rates than others and so tend to short-term thinking that promotes that kind of behavior, and certainly they have fewer reasons to try to live life as a non-addict. But I think the idea that morphine is no longer interesting and you can't become addicted when you live a stimulating life is wishful thinking.

Basics of Human Reinforcement

28 Post author: Yvain 06 July 2011 09:09PM

Today: some more concepts from reinforcement learning and some discussion on their applicability to human behavior.

For example: most humans do things even when they seem unlikely to result in delicious sugar water. Is this a violation of behaviorist principles?

No. For one thing, yesterday's post included a description of secondary reinforcers, those reinforcers which are not hard-coded evolutionary goods like food and sex, but which nevertheless have a conditioned association with good things. Money is the classic case of a secondary reinforcer among humans. Little colored rectangles are not naturally reinforcing, but from a very young age most humans learn that they can be used to buy pleasant things, like candy or toys or friends. Behaviorist-inspired experiments on humans often use money as a reward, and have yet to run into many experimental subjects whom it fails to motivate1.

Speaking of friends, status may be a primary reinforcer specific to social animals. I don't know if being able to literally feel reinforcement going on is a real thing, but I maintain I can feel the rush of reward when someone gives me a compliment. If that's too unscientific for you, consider studies in which monkeys will "exchange" sugary juice for the opportunity to look at pictures of high status monkeys, but demand extra juice in exchange for looking at pictures of low status monkeys.

Although certain cynics might consider money and status an exhaustive list, we may also add moral, aesthetic, and value-based considerations. Evolutionary psychology explains why these might exist and Bandura called some of them "internal reinforcement".

But more complicated reinforcers alone are not sufficient to bridge the gap between lever-pushing pigeons and human behavior. Humans have an ability to select for or against behaviors without trying them. For example: most of us would avoid going up to Mr. T and giving him the finger. But most of us have not personally tried this behavior and observed the consequences.

Is this the result of pure reason? No; the rational part of our mind is the part telling us that Mr. T is probably sixty years old by now and far too deep in the media spotlight to want to risk a scandal and jail time by beating up a random stranger. So where exactly is the reluctance coming from?


GENERALIZATION

Roko wrote in his post Ugh Fields that "your brain propagates psychological pain back to the earliest reliable stimulus for the punishment". This deserves more investigation.

Suppose you did go into a bar one night, see Mr. T, give him the finger, and get beaten up. What behavior would you avoid in the future based on this experience? The event itself does not immediately provide enough information to distinguish among "don't go into bars", "don't go out at night", "don't interact with people who have facial hair", and the correct answer "don't offend scary-looking people". This information has to come from your pre-existing model of reality, your brain's evolved background assumptions, and some clever guesswork.

Let's get back to the hilariously unethical experiments. Little Albert was an eight month old child who briefly starred in an experiment by behaviorist John Watson. Watson showed him a fuzzy white rat. Albert seemed to like the rat well enough. After Albert liking the rat had been confirmed, Watson showed him the rat again, but this time also played a very loud and scary noise; he repeated this intervention until, as expected, Albert was terrified of the white rat.

But it wasn't just fuzzy white rats Albert didn't like. Further investigation determined that Albert was also afraid of brown rabbits (fuzzy animal) and Santa Claus (fuzzy white beard). With his incipient powers of categorization, he had learned to associate punishment with a broad category of things vaguely clustered around fuzzy white rats.

B.F. Skinner had an even more interesting experiment that showed what happened when feedback of consequences went wrong. He put pigeons in a box that gave them rewards randomly. The pigeons ended up developing what he called "superstitions"; if a reward arrived by coincidence when a pigeon was tilting its head in a certain direction, the pigeon would continue tilting its head in that direction in the hope of gaining more rewards; when the reward randomly arrived, the pigeon took this as "justification" of its head-tilting and head-tilted even more2.

This provides one piece of the puzzle in the Mr. T question. None of us have ever given Mr. T the finger before. But we may have offended scary-looking people and had bad things happen to us, which our brains correctly generalize to "don't offend scary-looking people".

SOCIAL LEARNING

Or maybe not. Maybe you've never offended a scary-looking person before. what then?

Social learning theory is held up as opposed to behaviorism a lot, but it seems more like a natural extension of it. Humans and animals learn behaviors not just by being rewarded or punished themselves, but in observing whether a behavior is rewarded or punished in others.

Even if we ourselves have never offended scary-looking people, we have seen other people do so, or heard stories about people doing so, or watched people do so on TV.

At this point I have to mention my favorite social learning story ever, which also illustrates the pitfalls of trying to feedback consequences to their proximal cause. There has been some hand-wringing lately about children's TV shows and whether they lead to developmental problems in children. A study by Ostrov and Gentile cited in NurtureShock found the expected correlation between violent TV shows and physical aggression, but also found a an even stronger correlation between educational TV shows and so-called "relational aggression" - things like bullying, name-calling, and deliberate ostracism. The shows most strongly correlated with bad behavior were heart-warming educational programs intended to teach morality. Why?

The researchers theorize that the structure of these shows often involved a child committing an immoral action, the child looking cool and strong, and then at the end of the show the child eventually gets a comeuppance (think Harry Potter, where evil character Draco Malfoy is the coolest and most popular kid in Hogwarts and usually gets away with it, whereas supposedly sympathetic character Ron Weasley is at best a lovable loser who spends most of his time as the butt of Draco's jokes). The theory is that children are just not good enough at the whole feedback of conseqeunces thing to realize that the bully's comeuppance in the end is supposed to be the inevitable result of their evil ways. All they see is someone being a bully and then being treated as obviously popular and high-status.

Behavior is selection by consequences, and status is a strong reinforcer. If children see other children behaving as bullies and having high status, then all else being equal, they will be more likely to behave as bullies.

These two phenomena - feedback to categories and social learning - go part of the way to explaining the original question of how people have strong preferences for or against behaviors they've never tried before.

INTERNAL REINFORCEMENT

The phrase "internal reinforcement" would make good behaviorists cringe, seeing as it takes a perfectly good predictive model of behavior and tries to pin it on invisible mental phenomena.

But all reinforcement has to be at least a little internal; an animal wouldn't know that eating food was good and eating rocks was bad unless some internal structure knew to reinforce food-eating behavior but not rock-eating behavior. Some reinforcement seems even more internal than that; people may continue an activity solely because it makes them feel good about themselves.

This is not any more mysterious than eating behavior - the drive for food and the drive for status as measured in self-esteem are both perfectly legitimate biological drives, and it's not surprising that we have structures for noticing when we satisfy them and reinforcing the behavior involved - but it sure does sound less scientific.

STILL NOT GOOD ENOUGH

Much to the chagrin of behaviorists, all these mechanisms are still not sufficient to completely explain human behavior. Some cases - for example a patient who quits an enjoyable smoking habit because the doctor says it will cause cancer - may not fit any of these patterns. The patient may not previously have encountered any problems, personally or vicariously, with smoking or anything sufficiently similar to smoking to justify generalization, and positing internal reinforcement just moves the problem to another level.

Daniel Dennett speaks of

a sort of inner environment, in which tryouts can be safely executed - an inner something-or-other structured in such a way that the surrogate actions it favors are more often than not the very actions the real world would also bless, if they were actually performed. In short, the inner environment, whatever it is, must contain lots of information about the outer environment and its regularities. Nothing else (except magic) could provide preselection worth having.

There is some evidence for this sort of thing in certain cases: in experiments on fictive reinforcement, people who stayed out of a simulated rising stock market, thus breaking even when they could have won a lot of money, were found on MRI to have a reinforcement signal almost as if they were simulating the case in which they had entered the stock market and been reinforced for doing so.

But overall this idea involves too much magic and doesn't correspond to the way we really make decisions, either as perceived intuitively or as detected by most experiments. It also doesn't explain why we're so bad at being motivated by this sort of reinforcement: for example, since I know that heroin is really really enjoyable, why can't I become addicted to heroin just by thinking about it? And how come the overwhelming majority of patients don't quit smoking when their doctor tells them to do so, but people often do quit smoking after they've personally experienced the negative consequences (eg had their first heart attack)?

I am more favorable to the idea of a neural net model in which medical advice can forge a weak connection between the "smoking" pattern and the "cancer" pattern through cognition alone, separate from reinforcement processes but allowing such processes to propagate down it. Not a whole lot of motivational force can travel down such a weak link, blocking it from being effective against a strong desire to keep smoking. But I've got to admit that's a wild guess.

The important point, though, is that just as utility theory posits not just utility but expected utility, reinforcement learning posits not just reward but expected reward. Many processes by which we compute expected reward remain vague. Others have been explored in some detail. The next two posts will make up for the vagueness of this one by discussing some properties of the expected reward function.


FOOTNOTES:

1. Humans are not the only species that can become attracted to secondary reinforcers; monkeys have been successfully trained to use currency.

2: You can see the same effect at work in human athletes. If a certain behavior correlates with a winning streak, they will continue that behavior no matter how unlikely a causal link. But these athletes are curiosities precisely because people are so good at feeding back consequences to the correct stimulus.

Comment author: timtyler 06 July 2011 09:10:23AM *  3 points [-]

Just as evolving organisms are adaptation-executors and not fitness-maximizers, so minds are behavior-executors and not utility-maximizers.

This web site apparently has "a thing" about bashing utility-maximizers. Biologists use the concept of fitness maximisation as the fundamental unifying principle in biology. It builds animal brains, as best it can - so that they maximise the same function. As such, the idea deserves a hat tip - or you risk losing the baby with the bathwater.

I have a hypothesis about what is going on here: If humans don't maximise utility but machine intelligences do - and if maximisihg utility sometimes leads to utilitronium shockwave scenarios - then that makes machines look bad and scary. Making machines look bad and scary is a major theme around here.

Comment author: Yvain 06 July 2011 09:15:42AM 7 points [-]

This is the hat-tip to that idea. Evolution/organisms are so good at maximizing fitness that it's really tempting to think they're perfect at it, or at least perfectly efficient at it, which is something we have to be reminded again and again isn't true.

If it wasn't so close to being universally true, we wouldn't have to keep reminding people that it's occasionally false.

Comment author: jimmy 06 July 2011 07:38:14AM *  9 points [-]

Man, whoever told you about that must be well read ;-)

In all seriousness though, I would expect it to work for him, since the placebo effect seems to work for caffeine.

The "anti placebo effect" is the result of a simple classical conditioning. It looks like the strength is just some simple function of the relative timing of the CS and US. It 's an attempt to maintain homeostasis so you don't die.

The "regular placebo effect" is a different beast, and seems to have more or less full access to full cognitive capacity. The information about the contents of the pill gets fed 'down' to the "prior" input to your bayesian estimators. To the extent that the priors are strong and the data is weak, the final estimate sent back up can look a lot like the priors. This actually explains a ton of cool stuff, but that's for another comment/post.

The vast majority of examples of this "anti placebo effect" that I have read about involve injections (there was one story of oral intake). I'm really speculating here, but I think that short time frame things like injections would call up the "anti placebo" more strongly than "regular placebo". The first is that the other drug tolerance mechanisms can usually handle things when the onset is slow, but if the onset occurs in seconds, there has to be a quicker cognitive method to handle it. The second is that the signal to noise ratio is much higher with fast rise times which reduces the effect of the regular placebo effect, and increases the ease of learning the anti placebo effect.

Here are a couple sources on the classical conditioning effect on heroin tolerance.

Comment author: Yvain 06 July 2011 08:05:20AM 1 point [-]

Sorry not to credit you; I remembered it was someone at the LA meetup but not exactly who.

Comment author: Normal_Anomaly 04 July 2011 01:08:23AM 6 points [-]

Can you explain this quote? I don't understand what the "simplicity on this side of complexity" and the "simplicity on the other side of complexity" are. Does he mean naive opinions and well-thought-out opinions? Or folk theories and deep elegant true theories?

Comment author: Yvain 06 July 2011 02:39:03AM 5 points [-]

I think the simplicity on this side of complexity is naive theories that "just make sense" and the simplicity on the other side of complexity is mathematical elegance.

When one of the commenters in the Amanda Knox thread said yesterday that the probability has to be either 0 or 1 because either she did it or she didn't, that sounds simple. The mathematics of Bayesian probability are also simple, in that they can be derived from a few premises and explain a wide variety of disparate situations. But they're not the same sort of simplicity.

Comment author: FiftyTwo 06 July 2011 12:34:08AM 8 points [-]

Re reference 1. Another example of this working in humans is that apparently school children become conditioned to the class interval bell to the extent that they receive a boost of adrenaline.

Is it possible to deliberately establish such responses in oneself? For example if i wish to become more alert on demand could I pair stimulant use with another experience? I could see useful applications to this.

Comment author: Yvain 06 July 2011 02:19:35AM 21 points [-]

You might get the opposite result you expected.

We were talking about this at the last LA meetup in the context of place conditioning of drug reactions. Place conditioning occurs when you always take heroin (a drug that lowers your breathing rate) in the same place with the same people. You gradually develop a tolerance and take more and more heroin to get the same effects. Then one day you're on vacation and you take heroin in a different place with different people, and die.

The theory is that every time you're in your heroin spot with your heroin buddies (stimulus), your breath rate decreases and the nervous system has to raise the breath rate (response). Your nervous system gets pretty good at this, so even when you take large doses of heroin, you don't stop breathing. But if you take a large dose of heroin somewhere else, then your nervous system doesn't realize it has to raise the breath rate until too late, and the heroin decreases your breathing without any counter-efforts from the body and you die.

If you were to take a stimulant after playing a tone, your body would probably learn that your heart rate and various other metabolic parameters it likes to keep constant always go up after playing the tone, and as a result it would learn to lower heart rate etc. every time it hears the tone. When you play the tone without the stimulant, your body would lower metabolism and you wouldn't receive anything to counter that effect, so you'd end up less energetic and not more.

Comment author: lukeprog 09 June 2011 07:27:24AM 30 points [-]

This week on Facebook, Derek Sivers (founder of CD Baby) wrote that this article had more impact on him than anything else he read all year. He said: "Of all the articles I've read in the past 6 months, this one had the biggest impact on me."

Comment author: Yvain 05 July 2011 09:13:20PM 6 points [-]

I hadn't seen that! Thanks for bringing it up.

Comment author: MixedNuts 05 July 2011 06:57:09AM 4 points [-]

If you make the old mistake of confusing thoughts in general with analytic, reflective, verbal, serial internal monologue, I'm going to be sad.

Opaque processes decide to go to Subway tonight because they've heard it's low calorie, then they produce the verbal sentence "I should go to Subway tonight because it's low calorie"

I find this rather alien. Some processes are opaque, but that kind definitely isn't. Something (hunger, time, memory of previously made plans, whatever) triggers a reusable pick-a-sandwich-shop process; names and logos of nearby stores come up; associated emotions and concepts come up; weights associated to each shift - an image of those annoying health freaks who diet all the time upvotes "tasty" and downvotes "low calorie"; eventually they stabilize, create an image of myself going to Subway rather than somewhere else, and hand it over to motor control. If something gets stuck at any point, the process stops, a little alarm rings, and internal monologue turns to it to make it come unstuck. If not, there are no verbal thoughts at any point.

Comment author: Yvain 05 July 2011 08:59:47PM 2 points [-]

Probably time to start being sad; I'm mostly going to use "thoughts" that way. But I think what I'm talking about holds for any definition of "thought" where it's a mental activity accessible to the conscious mind.

I recognize different people use internal monologue to a different degree than others, but whether you decide with a monologue or with vague images of concepts, I think the core idea that these are attempts to turn subjective processes into objects for thought, usually so that you can weave a social narrative around them, remains true.

Basics of Animal Reinforcement

45 Yvain 05 July 2011 08:42PM

Behaviorism historically began with Pavlov's studies into classical conditioning. When dogs see food they naturally salivate. When Pavlov rang a bell before giving the dogs food, the dogs learned to associate the bell with the food and salivate even after they merely heard the bell . When Pavlov rang the bell a few times without providing food, the dogs stopped salivating, but when he added the food again it only took a single trial before the dogs "remembered" their previously conditioned salivation response1.

So much for classical conditioning. The real excitement starts at operant conditioning. Classical conditioning can only activate reflexive actions like salivation or sexual arousal; operant conditioning can produce entirely new behaviors and is most associated with the idea of "reinforcement learning".

Serious research into operant conditioning began with B.F. Skinner's work on pigeons. Stick a pigeon in a box with a lever and some associated machinery (a "Skinner box"2). The pigeon wanders around, does various things, and eventually hits the lever. Delicious sugar water squirts out. The pigeon continues wandering about and eventually hits the lever again. Another squirt of delicious sugar water. Eventually it percolates into its tiny pigeon brain that maybe pushing this lever makes sugar water squirt out. It starts pushing the lever more and more, each push continuing to convince it that yes, this is a good idea.

Consider a second, less lucky pigeon. It, too, wanders about in a box and eventually finds a lever. It pushes the lever and gets an electric shock. Eh, maybe it was a fluke. It pushes the lever again and gets another electric shock. It starts thinking "Maybe I should stop pressing that lever." The pigeon continues wandering about the box doing anything and everything other than pushing the shock lever.

The basic concept of operant conditioning is that an animal will repeat behaviors that give it reward, but avoid behaviors that give it punishment3.

Skinner distinguished between primary reinforcers and secondary reinforcers. A primary reinforcer is hard-coded: for example, food and sex are hard-coded rewards, pain and loud noises are hard-coded punishments. A primary reinforcer can be linked to a secondary reinforcer by classical conditioning. For example, if a clicker is clicked just before giving a dog a treat, the clicker itself will eventually become a way to reward the dog (as long as you don't use the unpaired clicker long enough for the conditioning to suffer extinction!)

Probably Skinner's most famous work on operant conditioning was his study of reinforcement schedules: that is, if pushing the lever only gives you reward some of the time, how obsessed will you become with pushing the lever?

Consider two basic types of reward: interval, in which pushing the lever gives a reward only once every t seconds - and ratio, in which pushing the lever gives a reward only once every x pushes.

Put a pigeon in a box with a lever programmed to only give rewards once an hour, and the pigeon will wise up pretty quickly. It may not have a perfect biological clock, but after somewhere around an hour, it will start pressing until it gets the reward and then give up for another hour or so. If it doesn't get its reward after an hour, the behavior will go extinct pretty quickly; it realizes the deal is off.

Put a pigeon in a box with a lever programmed to give one reward every one hundred presses, and again it will wise up. It will start pressing more on the lever when the reward is close (pigeons are better counters than you'd think!) and ease off after it obtains the reward. Again, if it doesn't get its reward after about a hundred presses, the behavior will become extinct pretty quickly.

To these two basic schedules of fixed reinforcement, Skinner added variable reinforcement: essentially the same but with a random factor built in. Instead of giving a reward once an hour, the pigeon may get a reward in a randomly chosen time between 30 and 90 minutes. Or instead of giving a reward every hundred presses, it might take somewhere between 50 and 150.

Put a pigeon in a box on variable interval schedule, and you'll get constant lever presses and good resistance to extinction.

Put a pigeon in a box with a variable ratio schedule and you get a situation one of my professors unscientifically but accurately described as "pure evil". The pigeon will become obsessed with pecking as much as possible, and really you can stop giving rewards at all after a while and the pigeon will never wise up.

Skinner was not the first person to place an animal in front of a lever that delivered reinforcement based on a variable ratio schedule. That honor goes to Charles Fey, inventor of the slot machine.

So it looks like some of this stuff has relevance for humans as well4. Tomorrow: more freshman psychology lecture material. Hooray!



FOOTNOTES

1. Of course, it's not really psychology unless you can think of an unethical yet hilarious application, so I refer you to Plaud and Martini's study in which slides of erotic stimuli (naked women) were paired with slides of non-erotic stimuli (penny jars) to give male experimental subjects a penny jar fetish; this supports a theory that uses chance pairing of sexual and non-sexual stimuli to explain normal fetish formation.

2. The bizarre rumor that B.F. Skinner raised his daughter in a Skinner box is completely false. The rumor that he marketed a child-rearing device called an "Heir Conditioner" is, remarkably, true.

3: In technical literature, behaviorists actually use four terms: positive reinforcement, positive punishment, negative reinforcement, and negative punishment. This is really confusing: "negative reinforcement" is actually a type of reward, behavior like going near wasps is "punished" even though we usually use "punishment" to mean deliberate human action, and all four terms can be summed up under the category "reinforcement" even though reinforcement is also sometimes used to mean "reward as opposed to punishment". I'm going to try to simplify things here by using "positive reinforcement" as a synonym for "reward" and "negative reinforcement" as a synonym for "punishment", same way the rest of the non-academic world does it.

4: Also relevant: checking HP:MoR for updates is variable interval reinforcement. You never know when an update's coming, but it doesn't come faster the more times you reload fanfiction.net. As predicted, even when Eliezer goes weeks without updating, the behavior continues to persist.

Comment author: RichardKennaway 05 July 2011 04:07:05PM 12 points [-]

I agree with you that behaviorism and PCT are different, which is why I don't understand why you're interpreting the robot as PCT and not behaviorist. From the program, it seems pretty clearly (STIMULUS: see blue -> RESPONSE: fire laser) to me.

Well, your robot example was an intuition pump constructed so as to be as close as possible to stimulus-response nature. If you consider something only slightly more complicated the distinction may become clearer: a room thermostat. Physically ripped out of its context, you can see it as a stimulus-response device. Temperature at sensor goes above threshold --> close a switch, temperature falls below threshold --> open the switch. You can set the temperature of the sensor to anything you like, and observe the resulting behaviour of the switch. Pure S-R.

In context, though, the thermostat has the effect of keeping the room temperature constant. You can no longer set the temperature of the sensor to anything you like. Put a candle near it, and the temperature of the rest of the room will fall while the sensor remains at a constant temperature. Use a strong enough heat source or cold source, and you will be able to overwhelm the control system's efforts to maintain a constant temperature, but this fails to tell you anything about how the control system works normally. Do the analogous thing to a living organism and you either kill it or put it under such stress that whatever you observe is unlikely to tell you much about its normal operation -- and biology and psychology should be about how organisms work, not how they fail under torture.

Did you know that lab rats are normally starved until they have lost 20% of their free-feeding weight, before using them in behavioural experiments?

Here's a general block diagram of a control system. The controller is the part above the dotted line and its environment the part below (what would be called the plant in an industrial context). R = reference, P = perception, O = output, D = disturbance (everything in the environment besides O that affects the perception). I have deliberately drawn this to look symmetrical, but the contents of those two boxes makes its functioning asymmetrical. P remains close to R, but O and D need have no visible relationship at all.

 R |
|
V
+-------+
| |
+--->| |----+
| | | |
^ +-------+ v
| |
....... P | ............... | O .......
| |
^ +-------+ v
| | | |
+----| |<---+
| |
+-------+
^
|
D |

When you are dealing with a living organism, R is somewhere inside it. You probably cannot measure it even if you know it exist. (E.g. just what and where, physically, is the set point for deep body temperature in a mammal? Not an easy question to answer.) You may or may not know what P is -- what the organism is actually sensing. It is important to realise that when you perform an experiment on an animal, you have no way of setting P. All you can do is create a disturbance D that may influence P. D, from a behavioural point of view, is the "stimulus" and O, the creature's action on its environment, is the "response". the behaviourist description of the situation is this:

 +-------+
D | | O
----->| |----->
| |
+-------+

This is simply wrong. The system does not work like that and cannot be understood like that. It may look as if D causes O, but that is like thinking that a candle put in a certain place chills the room, a fact that will seem mysterious and paradoxical when you do not know that the thermostat is present, and will only be explained by discovering the actual mechanism, discarding the second diagram in favour of the first. No amount of data collection will help until one has made that change. This is why correlations are so lamentably low in psychological experiments.

Do you have GChat or any kind of instant messenger?

No, I've never used any of those systems. I prefer a medium in which I can take my time to work out exactly what I want to say.

Comment author: Yvain 05 July 2011 07:39:11PM *  8 points [-]

Okay, we agree that the simple robot described here is behaviorist and the thermostat is PCT. And I certainly see where you're coming from with the rats being PCT because hunger only works as a motivator if you're hungry. But I do have a few questions:

  1. There are some things behaviorism can explain pretty well that I don't know how to model in PCT. For example, consider heroin addiction. An animal can go its whole life not wanting heroin until it's exposed to some. Then suddenly heroin becomes extraordinarily motivating and it will preferentially choose shots of heroin to food, water, or almost anything else. What is the PCT explanation of that?

  2. I'm not entirely sure which correlation studies you're talking about here; most psych studies I read are done in an RCT type design and so use p-values rather than r-values; they can easily end up with p < .001 if they get a large sample and a good hypothesis. Some social psych studies work off of correlations (eg correlation between being observer-rated attractiveness and observer-rated competence at a skill); correlations are "lamentably low" in social psychology because high level processes (like opinion formation, social interaction, etc.) have a lot of noise. Are there any PCT studies of these sorts of processes (not simple motor coordination problems) that have any higher correlation than standard models do? Any with even the same level of correlation?

  3. What's the difference between control theory and stimulus-response in a context? For example, if we use a simplified version of hunger in which the hormone leptin is produced in response to hunger and the hormone ghrelin is produced in response to satiety, we can explain this in two ways: the body is trying to PCT itself to the perfect balance of leptin and ghrelin, or in the context of the stimulus leptin the response of eating is rewarded and in the context of the stimulus ghrelin the response of eating is punished. Are these the same theory, or are there experiments that would distinguish between them? Do you know of any?

  4. Does PCT still need reinforcement learning to explain why animals use some strategies and not others to achieve equilibrium? For example, when a rat in a Skinner box is hungry (ie its satiety variable has deviated in the direction of hunger), and then it presses a lever and gets a food pellet and its satiety variable goes back to its reference range, would PCTists explain that as getting rewarded for pressing the lever and expect it to press the lever again next time it's hungry?

The Blue-Minimizing Robot

162 Yvain 04 July 2011 10:26PM

Imagine a robot with a turret-mounted camera and laser. Each moment, it is programmed to move forward a certain distance and perform a sweep with its camera. As it sweeps, the robot continuously analyzes the average RGB value of the pixels in the camera image; if the blue component passes a certain threshold, the robot stops, fires its laser at the part of the world corresponding to the blue area in the camera image, and then continues on its way.

Watching the robot's behavior, we would conclude that this is a robot that destroys blue objects. Maybe it is a surgical robot that destroys cancer cells marked by a blue dye; maybe it was built by the Department of Homeland Security to fight a group of terrorists who wear blue uniforms. Whatever. The point is that we would analyze this robot in terms of its goals, and in those terms we would be tempted to call this robot a blue-minimizer: a machine that exists solely to reduce the amount of blue objects in the world.

Suppose the robot had human level intelligence in some side module, but no access to its own source code; that it could learn about itself only through observing its own actions. The robot might come to the same conclusions we did: that it is a blue-minimizer, set upon a holy quest to rid the world of the scourge of blue objects.

But now stick the robot in a room with a hologram projector. The hologram projector (which is itself gray) projects a hologram of a blue object five meters in front of it. The robot's camera detects the projector, but its RGB value is harmless and the robot does not fire. Then the robot's camera detects the blue hologram and zaps it. We arrange for the robot to enter this room several times, and each time it ignores the projector and zaps the hologram, without effect.

Here the robot is failing at its goal of being a blue-minimizer. The right way to reduce the amount of blue in the universe is to destroy the projector; instead its beams flit harmlessly through the hologram.

Again, give the robot human level intelligence. Teach it exactly what a hologram projector is and how it works. Now what happens? Exactly the same thing - the robot executes its code, which says to scan the room until its camera registers blue, then shoot its laser.

In fact, there are many ways to subvert this robot. What if we put a lens over its camera which inverts the image, so that white appears as black, red as green, blue as yellow, and so on? The robot will not shoot us with its laser to prevent such a violation (unless we happen to be wearing blue clothes when we approach) - its entire program was detailed in the first paragraph, and there's nothing about resisting lens alterations. Nor will the robot correct itself and shoot only at objects that appear yellow - its entire program was detailed in the first paragraph, and there's nothing about correcting its program for new lenses. The robot will continue to zap objects that register a blue RGB value; but now it'll be shooting at anything that is yellow.

The human-level intelligence version of the robot will notice its vision has been inverted. It will know it is shooting yellow objects. It will know it is failing at its original goal of blue-minimization. And maybe if it had previously decided it was on a holy quest to rid the world of blue, it will be deeply horrified and ashamed of its actions. It will wonder why it has suddenly started to deviate from this quest, and why it just can't work up the will to destroy blue objects anymore.

The robot goes to Quirinus Quirrell, who explains that robots don't really care about minimizing the color blue. They only care about status and power, and pretend to care about minimizing blue in order to impress potential allies.

The robot goes to Robin Hanson, who explains that there are really multiple agents within the robot. One of them wants to minimize the color blue, the other wants to minimize the color yellow. Maybe the two of them can make peace, and agree to minimize yellow one day and blue the next?

The robot goes to Anna Salamon, who explains that robots are not automatically strategic, and that if it wants to achieve its goal it will have to learn special techniques to keep focus on it.

I think all of these explanations hold part of the puzzle, but that the most fundamental explanation is that the mistake began as soon as we started calling it a "blue-minimizing robot". This is not because its utility function doesn't exactly correspond to blue-minimization: even if we try to assign it a ponderous function like "minimize the color represented as blue within your current visual system, except in the case of holograms" it will be a case of overfitting a curve. The robot is not maximizing or minimizing anything. It does exactly what it says in its program: find something that appears blue and shoot it with a laser. If its human handlers (or itself) want to interpret that as goal directed behavior, well, that's their problem.

It may be that the robot was created to achieve a specific goal. It may be that the Department of Homeland Security programmed it to attack blue-uniformed terrorists who had no access to hologram projectors or inversion lenses. But to assign the goal of "blue minimization" to the robot is a confusion of levels: this was a goal of the Department of Homeland Security, which became a lost purpose as soon as it was represented in the form of code.

The robot is a behavior-executor, not a utility-maximizer.

In the rest of this sequence, I want to expand upon this idea. I'll start by discussing some of the foundations of behaviorism, one of the earliest theories to treat people as behavior-executors. I'll go into some of the implications for the "easy problem" of consciousness and philosophy of mind. I'll very briefly discuss the philosophical debate around eliminativism and a few eliminativist schools. Then I'll go into why we feel like we have goals and preferences and what to do about them.

Comment author: Perplexed 04 July 2011 09:16:18PM 2 points [-]

I take this to be an elliptical way of suggesting that Yvain is offering a false dichotomy in suggesting a choice between the notion of thoughts being in control of the processes determining behavior and the notion of thoughts being a byproduct of those processes.

I agree. Thoughts are at one with (are a subset of) the processes that determine behavior.

Comment author: Yvain 04 July 2011 10:09:59PM *  2 points [-]

I'm not so sure. Using the analogy of a computer program, we could think of thoughts either as like the lines of code in the program (in which case they're at one with, or in control of, the processes generating behavior, depending on how you want to look at it) or you could think of thoughts as like the status messages that print "Reticulating splines" or "50% complete" to the screen, in which case they're byproducts of those processes (very specific, unnatural byproducts, to boot).

My view is closer to the latter; they're a way of allowing the brain to make inferences about its own behavior and to communicate those inferences. Opaque processes decide to go to Subway tonight because they've heard it's low calorie, then they produce the verbal sentence "I should go to Subway tonight because it's low calorie", and then when your friend asks you why you went to Subway, you say "Because it's low calorie").

The tendency of thoughts to appear in a conversational phrasing ("I think I'll go to Subway tonight") rather than something like "Dear Broca's Area - Please be informed that we are going to Subway tonight, and adjust your verbal behavior accordingly - yours sincerely, the prefrontal cortex" is a byproduct of their use in conversation, not their internal function.

Right now I'm just asserting that this is a possibility and that it's distinct from thoughts being part of the decision-making structure. I'll try to give some evidence for it later.

Comment author: lukeprog 04 July 2011 09:40:06PM 2 points [-]

Yes, 'folk psychology' (beliefs+desires=intentional action) is a compelling theory because it works so successfully in everyday social interactions, but I'm wondering if Yvain has something more planned for why it feels (inside) like we have goals. My guess is that it's because we use the same folk theory to explain our own behavior as to explain others' behavior, but I'm wondering if Yvain has something else in mind.

Comment author: Yvain 04 July 2011 09:42:08PM 1 point [-]

Nope, pretty much that. I'll be presenting a few studies to justify it, but I'm sure you've seen them before.

Comment author: Yvain 04 July 2011 09:04:08PM 11 points [-]

I can't access the article and I don't have time for a podcast, but the obvious question is "Why wouldn't all these same reasons apply to a chimp?"

There's pretty good evidence that there are alternate brain designs which can at least gain us 10 or 20 IQ points. There's a gene for promoting nerve growth (can't remember the name) which is associated with a higher IQ and a higher rate of cancer; in most people it's been selected against because of the cancer risk, but that suggests a very different reason why our brains are limited to their current power than Fox brings up. Gaucher's Disease is another deadly condition linked to genes which can increase IQ in people who don't have the disease (and increase it even more in people who have the disease but live with it).

So I would agree with him that there are probably biological reasons our IQ isn't higher, but I think they're more likely to involve disease than neural architecture, and that we can probably evolve ways around those diseases pretty quickly on an evolutionary scale.

Comment author: RichardKennaway 04 July 2011 07:59:10PM 1 point [-]

So are you saying that any program can be modeled by PCT better than by looking at the program itself, or that although this particular robot isn't PCT, a hypothetical robot that was more reflective of real human behavior would be?

I am saying that this particular robot (without the add-on human module) is a control system. It consists of nothing more than that single control system. It contains no representation of any part of itself. It does not reflect on its nature, or try to find other ways of achieving its goal.

The hierarchical arrangement of control systems that HPCT (Hierarchical PCT) ascribes to humans and other living organisms, is more complex. Humans have goals that are instrumental towards other goals, and which are discarded as soon as they become ineffective for those higher-level goals.

As for goals, if I understand your definition correctly, even a behaviorist system could be said to have goals (if you reinforce it every time it pulls the lever, then its new goal will be to pull a lever). If that's your definition, I agree that this robot has goals, and I would rephrase my thesis as being that those goals are not context-independent and reflective.

Behaviourism is a whole other can of worms. It models living organisms as stimulus-response systems, in which outputs are determined by perceptions. PCT is the opposite: perceptions are determined by outputs.

Comment author: Yvain 04 July 2011 08:43:13PM *  6 points [-]

I agree with you that behaviorism and PCT are different, which is why I don't understand why you're interpreting the robot as PCT and not behaviorist. From the program, it seems pretty clearly (STIMULUS: see blue -> RESPONSE: fire laser) to me.

Do you have GChat or any kind of instant messenger? I feel like real-time discussion might be helpful here, because I'm still not getting it.

Behaviorism: Beware Anthropomorphizing Humans

53 Yvain 04 July 2011 08:40PM

Related to: The Comedy of Behaviorism

Behaviorism's gotten a bad rap.

It's gone down in history as the school founded upon the idea that there's no such thing as mental phenomena or cognitive processing, and if there are we can't ever know anything about them, and if we can I don't want to know about it, and if you tell me I will put my fingers in my ears and whistle, and SHUT UP SHUT UP I CAN'T HEAR YOU.

Actually it was more subtle.

The movement did begin with a variation on that principle for historical reasons. John Watson began his work thirty years before the first computer. Information processing still looked like magic; most scientists didn't realize that reductionist accounts of information processing were even possible. Neurons were still "the thing that Spanish guy keeps talking about". Today we discuss the brain by analogy to computers; in Watson's day, they discussed the brain by analogy to their own most advanced technology, mechanical devices. Today we talk about looking for mental programs and subroutines; they sought its gears and levers instead. And just as today many philosophers dismiss consciousness as an epiphenomenon of information processing because computers don't seem to be conscious, so Watson dismissed all mental states as an epiphenomenon of mechanical processing because mechanical devices didn't have mental states.

As science advanced, and as it picked up glimpses of cognition from the Stroop effect and early priming experiments, behaviorism became more sophisticated. Maybe its pinnacle of subtlety came with B.F. Skinner's "radical behaviorism" movement, which accepted inner mental life (which Skinner called "mental behavior") and sought to explain it.

If Skinner was willing to acknowledge inner life, why do we still call his theory behaviorist? It's hard and not especially profitable to define "behaviorism", but if I had to try I'd say it is a methodology that doesn't consider mental phenomena useful as a fundamental level of explanation. So if we want to know why Wanda runs away from a wasp, saying "because her previous encounters wasps have been negatively reinforced" is more useful than "because she felt scared".

And if Wanda herself says "No, I ran away because I felt scared," we shouldn't be especially interested in her opinion: she has privileged access to a certain type of output of the process generating her behavior, but not to the process itself.

Imagine the better behaviorists, if you like, as playing a worldwide half-century long game of Rationalist Taboo, in which you're no longer allowed to use words like "want", "feel", "hope", or "decide". It's overwhelmingly tempting to fake-explain psychology using non-technical non-explanations like "Oh, she just acts that way because she has an overly emotional personality" and so the whole school just promised themselves to root out that way of thinking.

Although the witticism that behaviorism scrupulously avoids anthropomorphizing humans was intended as a jab at the theory, I think it touches on something pretty important. Just as normal anthropomorphism - "it only snows in winter because the snow prefers cold weather", acts as a curiosity-stopper and discourages technical explanation of the behavior, so using mental language to explain the human mind equally halts the discussion without further investigation.

This idea of Rationalist Taboo also explains B.F. Skinner's "mental behavior" loophole. When he discusses thoughts as mental behavior, he's not using them as explanations for other things - not taking the easy way out and saying "The reason I stayed in tonight is because, after thinking about it, I decided I didn't want to go to dinner". He's taking an extra burden upon himself, trying to come up with explanations for thoughts as well as actions.

Behaviorism became less popular in the 1950s after clever experimental protocols allowed more direct measurement of what happens inside the mind, making its taboo on mental occurrences unnecessary and restrictive. Although the philosophical commitments involved became obsolete, the scientific findings remain as valuable as ever. They have entered into the new paradigm as "reinforcement learning", a process widely believed to underlie many diverse mental subsystems all the way from motor coordination to social behavior.

Although reinforcement learning is almost universally known, Skinner's philosophical context for the process is not. He believed that the Darwinian evolution of organisms was just one instance of a wider principle called "selection by consequences", the most successful optimization process in the history of the universe. Evolution can successfully design permanent features of an organism like its skin, claws, and eyes. But it is too slow to fully optimize an organism's behavior, and too large-grained to produce complex behavior on its own. It is is especially too slow and large-grained to produce human-level behavior: citing my sources in MLA format is an important skill, and I don't want to have to wait until ten generations of my ancestors have perished for citing their sources incorrectly before I can do it right.

So evolution conjured up a mini-evolution to serve it. Reinforcement learning is evolution writ small; behaviors propagate or die out based on their consequences to reinforcement in a mind, just as mutations propagate or die out based on their consequences to reproduction in an organism. In the behaviorist model, our mind is not an agent, but a flourishing ecosystem of behaviors both physical and mental, all scrabbling for supremacy and mutating into more effective versions of themselves.

Just as evolving organisms are adaptation-executors and not fitness-maximizers, so minds are behavior-executors and not utility-maximizers. This returns us to the case of the blue-minimizing robot, which executed its program without any representation of a "goal". Behaviorism holds out the prospect of an explanation of human behavior based on similar lines.

Despite its subsumption by the cognitive paradigm, behaviorism continues to hold a special place because of its association with reinforcement learning, as well as its uses in industrial psychology, applied psychology, and various successful therapies including the famous CBT. It's also one of the major inspirations for connectionism, a more modern and exciting eliminativist model which we'll return to later.

This sequence will continue by exploring some of the basics of reinforcement learning in the behaviorist paradigm, and then get into more controversial applications of the theory to explain previously mysterious human behaviors.

Comment author: Kaj_Sotala 04 July 2011 01:19:43PM *  10 points [-]

Ah, excellent. This post comes at a great time. A few weeks ago, I talked with someone who remarked that although decision theory speaks in terms of preferences and information being separate, trying to apply that into humans is fitting the data to the theory. He was of the opinion that humans don't really have preferences in the decision theoretic sense of the word. Pondering that claim, I came to the conclusion that he's right, and have started to increasingly suspect that CEV-like plans to figure out the "ultimate" preferences of people are somewhat misguided. Our preferences are probably hopelessly path-, situation- and information-dependent. Which is not to say that CEV would be entirely pointless - even if the vast majority of our "preferences" would never converge, there might be some that did. And of course, CEV would still be worth trying, just to make sure I'm not horribly mistaken on this.

The ease at which I accepted the claim "humans don't have preferences" makes me suspect that I've myself had a subconscious intuition to that effect for a long time, which was probably partially responsible for an unresolved disagreement between me and Vladimir Nesov earlier.

I'll be curious to hear what you have to say.

Comment author: Yvain 04 July 2011 07:04:21PM 0 points [-]

At the end of this, I'm going to try to argue that something like CEV is still justified. Before I started thinking it through I was hoping that taking an eliminativist view of preferences to its conclusion would help tie up the loopholes in CEV, and so far it hasn't done that for me, but it hasn't made it any harder either.

Comment author: RichardKennaway 04 July 2011 12:12:49PM *  12 points [-]

When you say the robot has a "different goal", I'm not sure what you mean. What is the robot's goal? To follow the program detailed in the first paragraph?

The robot's goal is not to follow its own program. The program is simply what the robot does. In the environment it is designed to operate in, what it does is destroy blue objects. In the vocabulary of control theory, the controlled variable is the number of blue objects, the reference value is zero, the difference between the two is the error, firing the laser is the action is takes when the error is positive, and the action has the effect of reducing the error. The goal, as with any control system, is to keep the error at zero. It does not have an additional goal of being the best destroyer of blue objects possible. Its designers might have that goal, but if so, that goal is in the designers, not in the system they have designed.

In an environment containing blue objects invulnerable to laser fire, the robot will fail to control the number of blue objects at zero. That does not make it not a control system, just a control system encountering disturbances it is unable to control. To ask whether it is still a control system veers into a purely verbal argument, like asking whether a table is still a table if one leg has broken off and it cannot stand upright.

People are more complex. They have (according to PCT) a large hierarchy of control systems (very broad, but less than a dozen levels deep), in which the reference signal for each controller is set by the output signals of higher level controllers. (At the top, reference signals are presumably hard-wired, and at the bottom, output signals go to organs not made of neurons -- muscles, mainly.) In addition, the hierarchy is subject to reorganisation and other forms of adaptation. The adaptations present to consciousness are the ability to think about our goals, consider whether we are doing the best things to achieve them, and change what we are doing. The robot in the example cannot do this.

You might be thinking of "goal" as meaning this sort of conscious, reflective, adaptive attempt to achieve what we "really" want, but I find that too large and fuzzy a concept. It leads into a morass of talk about our "real" goals vs. the goals we think we have, self-reflexive decision theory, extreme thought experiments, and so on. A real science of living things has to start smaller, with theories and observations that can be demonstrated as surely and reproducibly as the motion of balls rolling down inclined planes.

(ETA: When the neuroscience fails to discover this huge complex thing that never carved reality at the joints in the first place, people respond by saying it doesn't exist, that it went the way of kobolds rather than rainbows.)

Maybe you're also thinking of this robot's program as a plain stimulus-response system, as in the behaviourist view of living systems. But what makes it a control system is the environment it is embedded in, an environment in which shooting at blue objects destroys them.

If goals reduce to a program like the robot's in any way, it's in the way that Einsteinian mechanics "reduce" to Newtonian mechanics - giving good results in most cases but being fundamentally different and making different predictions on border cases.

If I replace "program" by "behaviourism", then I would say that it is behaviourism that is explained away by PCT.

Comment author: Yvain 04 July 2011 07:01:14PM *  8 points [-]

Now I'm very confused. I understand that you think humans are PCT systems and that you have some justifications for that. But unlike humans, we know exactly what motivates this robot (the program in the first paragraph) and it doesn't contain a controlled variable corresponding to the number of blue objects, or anything else that sounds PCT.

So are you saying that any program can be modeled by PCT better than by looking at the program itself, or that although this particular robot isn't PCT, a hypothetical robot that was more reflective of real human behavior would be?

As for goals, if I understand your definition correctly, even a behaviorist system could be said to have goals (if you reinforce it every time it pulls the lever, then its new goal will be to pull a lever). If that's your definition, I agree that this robot has goals, and I would rephrase my thesis as being that those goals are not context-independent and reflective.

Comment author: RichardKennaway 04 July 2011 12:19:00PM 0 points [-]

Richard's PCT-based definition of goal is very different from mine, and although it's easily applicable to things like controlling eye movements, it doesn't have the same properties as the philosophical definition of "goal", the one that's applicable when you're reading all the SIAI work about AI goals and goal-directed behavior and such.

Can you spell out the philosophical definition? My previous comment, which I posted before reading this, made only a vague guess at the concept you had in mind: "this sort of conscious, reflective, adaptive attempt to achieve what we 'really' want".

Comment author: Yvain 04 July 2011 06:54:38PM 2 points [-]

I think we agree, especially when you use the word "reflective". As opposed to, say, a reflex, which is an unconscious, nonreflective effort to acheive something which evolution or our designers decided to "want" for us. When the robot's reflection that shooting the hologram projector instead of the hologram fails to motivate it to do so, I start doubting its behaviors are goal-driven, and suspecting they're reflexive.

Comment author: RichardKennaway 04 July 2011 02:31:21PM 8 points [-]

this is all a metaphor for people

One that presents consciousness as an epiphenomenon. In the version of the robot that has human intelligence, you describe it as bolted on, experiencing the robot's actions but having no causal influence on them, an impotent spectator.

Are your projected postings going to justify this hypothesis?

Comment author: Yvain 04 July 2011 06:50:22PM 3 points [-]

I hope so. Let's see.

Comment author: pjeby 04 July 2011 12:40:09AM 5 points [-]

What is the robot's goal? To follow the program detailed in the first paragraph?

I suspect Richard would say that the robot's goal is minimizing its perception of blue. That's the PCT perspective on the behavior of biological systems in such scenarios.

However, I'm not sure this description actually applies to the robot, since the program was specified as "scan and shoot", not "notice when there's too much blue and get rid of it.". In observed biological systems, goals are typically expressed as perception-based negative feedback loops implemented in hardware, rather than purely rote programs OR high-level software algorithms. But without more details of the robot's design, it's hard to say whether it really meets the PCT criterion for goals.

Of course, from a certain perspective, you could say at a high level that the robot's behavior is as if it had a goal of minimizing its perception of blue. But as your post points out, this idea is in the mind of the beholder, not in the robot. I would go further as to say that all such labeling of things as goals occurs in the minds of observers, regardless of how complex or simple the biological, mechanical, electronic, or other source of behavior is.

Comment author: Yvain 04 July 2011 06:22:19AM *  4 points [-]

Although I find PCT intriguing, all the examples of it I've found have been about simple motor tasks. I can take a guess at how you might use the Method of Levels to explain larger-level decisions like which candidate to vote for, or whether to take more heroin, but it seems hokey, I haven't seen any reputable studies conducted at this level (except one, which claimed to have found against it) and the theory seems philosophically opposed to conducting them (they claim that "statistical tests are of no use in the study of living control systems", which raises a red flag large enough to cover a small city)

I've found behaviorism much more useful for modeling the things I want to model; I've read the PCT arguments against behaviorism and they seem ill-founded - for example, they note that animals sometimes auto-learn and behaviorist methodological insistence on external stimuli shouldn't allow that, but once we relax the methodological restrictions, this seems to be a case of surprise serving the same function as negative reinforcement, something which is so well understood that neuroscientists can even point to the exact neurons in charge of it.

Richard's PCT-based definition of goal is very different from mine, and although it's easily applicable to things like controlling eye movements, it doesn't have the same properties as the philosophical definition of "goal", the one that's applicable when you're reading all the SIAI work about AI goals and goal-directed behavior and such.

By my definition of goal, if the robot's goal were to minimize its perception of blue, it would shoot the laser exactly once - at its own visual apparatus - then remain immobile until turned off.

Comment author: endoself 04 July 2011 12:55:40AM *  6 points [-]

Why does the human-level intelligence component of the robot care about blue? It seems to me that it is mistaken in doing so. If my motor cortex was replaced by this robot's program, I would not conclude that I had suddenly started to only care about blue, I would conclude that I had lost control of my motor cortex. I don't see how it makes any difference that the robot always had it actions controlled by the blue-minimizing program. If I were the robot then, upon being informed about my design, I would conclude that I did not really care about blue. My human-level intelligence is the part that is me and therefore contains my preferences, not my motor cortex.

Comment author: Yvain 04 July 2011 06:13:33AM *  17 points [-]

If my motor cortex was replaced by this robot's program, I would not conclude that I had suddenly started to only care about blue, I would conclude that I had lost control of my motor cortex.

I predict this would not happen the way you anticipate, at least for some ways to cash out 'taking control of your motor cortex'. For example, when a neurosurgeon uses a probe to stimulate a part of the motor cortex responsible for moving the arm, and eir patient's arm moves, and the neurosurgeon asks the patient why ey moved eir arm, the patient often replies something like "I had an itch", "it was uncomfortable in that position", or "What, I'm not allowed to move my arm now without getting grilled on it?"

Or in certain forms of motor cortex damage in which patients can't move their arm, they explain it by saying "I could move my arm right now, I just don't feel like it" or "That's not even my real arm, how could you expect me to move that?".

Although I won't get there in a while, part of my thesis for this sequence is that we infer our opinions from our behaviors, although it's probably more accurate to say that our behaviors feed back to the same processes that generate our opinions and can alter them. If this is true, then there are probably very subtle ways of taking control of your motor cortex that would leave your speech centers making justifications for whatever you did.

Comment author: Psy-Kosh 04 July 2011 01:19:25AM 4 points [-]

Does that "human level intelligence module" have any ability to actually control the robot's actions, or just to passively observe and then ask "why did I do that?" What're the rules of the game, as such, here?

Comment author: Yvain 04 July 2011 06:02:50AM 3 points [-]

I don't think it's saying anything too shocking to admit this is all a metaphor for people; I'm going to be pushing the view that people's thoughts and words are a byproduct of the processes that determine behavior rather than directly controlling them. I anticipate providing at least a partial answer to your question in about two weeks; if that doesn't satisfy you, let me know and we can talk about it then.

Comment author: Yvain 03 July 2011 11:07:00PM *  11 points [-]

"I would not give a farthing for the simplicity on this side of complexity, but I would give my life for the simplicity on the other side of complexity."

-- Oliver Wendell Holmes (quoted by Venkatesh Rao; thanks to InquilineKea)

Comment author: fubarobfusco 03 July 2011 08:30:04PM 9 points [-]

we make outrageously bold claims about getting smarter and richer and sexier

I'd like to know where all this LW-boasting is going on. I don't think I hear it at the meetups in Mountain View, but maybe I've been missing something.

Comment author: Yvain 03 July 2011 10:30:57PM *  25 points [-]

Darnit, I don't like being vague and I also don't like pointing to specific people and saying "YOU! YOU SOUND CULTISH!" so I'm going to have a hard time answering this question in a satisfying way, but...

Lots of people are looking into things like nootropics/intelligence amplification, entrepreneurship, and pick-up artistry. And this is great. What gives me the creepy vibe is when they say (more on the site than at meetups) "And of course, we'll succeed at these much faster than other people have, even though they are professionals in this field, because we're Rationalists and they weren't." Anything involving the words "winning", "awesomeness", or gratuitous overuse of community identification terms like "primate" or "utility".

Trying to look for examples, I notice it is a smaller proportion of things than I originally thought and I'm probably biased toward overcounting them, which makes sense since in order to justify my belonging to a slightly creepy group I need to exaggerate my opposition to the group's creepiness.

Comment author: RichardKennaway 03 July 2011 09:50:23PM *  22 points [-]

I'll be interested to see where you go with this, but it seems to me that saying, "look, this is the program the robot runs, therefore it doesn't really have a goal", is exactly like saying "look, it's made of atoms, therefore it doesn't really have a goal".

Goals are precisely explained (like rainbows), and not explained away (like kobolds), as the controlled variables of control systems. This robot is such a system. The hypothetical goal of its designers at the Department of Homeland Security is also a goal. That does not make the robot's goal not a goal; it just makes it a different goal.

We feel like we have goals and preferences because we do, in fact, have goals and preferences, and we not only have them, but we are also aware of having them. The robot is not aware of having the goal that it has. It merely has it.

Comment author: Yvain 03 July 2011 10:09:12PM *  21 points [-]

First of all, your control theory work was...not exactly what started me thinking along these lines, but what made it click when I realized the lines I had been thinking along were similar to the ones I had read about in one of your introductory posts about performing complex behaviors without representations. So thank you.

Second - When you say the robot has a "different goal", I'm not sure what you mean. What is the robot's goal? To follow the program detailed in the first paragraph?

Let's say Robot-1 genuinely has the goal to kill terrorists. If a hacker were to try to change its programming to "make automobiles" instead, Robot-1 would do anything it could to thwart the hacker; its goal is to kill terrorists, and letting a hacker change its goal would mean more terrorists get left alive. This sort of stability, in which the preference remains a preference regardless of context are characteristic of my definition of "goal".

This "blue-minimizing robot" won't display that kind of behavior. It doesn't thwart the person who places a color inversion lens on it (even though that thwarts its stated goal of "minimizing blue"), and it wouldn't try to take the color inversion lens off even if it had a manipulator arm. Even if you claim its goal is just to "follow its program", it wouldn't use its laser to stop someone walking up to it and changing its program, which means its program no longer got followed.

This isn't just a reduction of a goal to a program: predicting the robot's goal-based behavior and its program-based behavior give different results.

If goals reduce to a program like the robot's in any way, it's in the way that Einsteinian mechanics "reduce" to Newtonian mechanics - giving good results in most cases but being fundamentally different and making different predictions on border cases. Because there are other programs that goals do reduce to, like the previously mentioned Robot-1, I don't think it's appropriate to call what the blue-minimizer is doing a "goal".

If you still disagree, can you say exactly what goal you think the robot is pursuing, so I can examine your argument in more detail?

Comment author: endoself 03 July 2011 05:24:28PM *  5 points [-]

I'm not sure I'd read much into "horny ladies" abounding in fiction (or joke books).

They do reflect the culture of the time. A illustrative example is the play Lysistrata, where the plot is based on women denying sex to men as a punishment. This was considered hilarious because people though it could never happen. Now, cultural expectations have changed and it happened.

In response to comment by endoself on Gender and Libido
Comment author: Yvain 03 July 2011 07:38:01PM 3 points [-]

Also addressed directly in the myth of Tiresias

Comment author: Yvain 03 July 2011 08:31:44AM *  31 points [-]

The advice, the tone, the vibe 'feels' wrong, somehow. If you forced me to use more precise language, I might say that, for several years now, I have kept a variety of procedural heuristics running in the background that help me ferret out bullshit, partisanship, wishful thinking, and other unsound debating tactics -- and important content on this website manages to trigger most of them.

To come up with a theory on the fly, maybe there are two modes of expansion for a group: by providing some service, and by sheer memetic virulence. One memetic virulence strategy operates by making outlandish promises that subscribing to it will make you smarter, richer, more successful, more attractive to the opposite sex, and just plain superior to other people - and then doing it in a way that can't obviously be proven wrong. This strategy usually involves people with loads of positive affect going around telling people how great their group is and how they need to join.

As a memetic defense strategy, people learn to identify this kind of spread and to shun groups that display its features. From the inside, this strategy manifests as a creepy feeling.

LW members have lots of positive affect around LW and express it all the time, the group seems to be growing without providing any services (eg no one's worried when a drama club grows, because they go and put on dramas, but it's not clear what our group is doing besides talking about how great we are), and we make outrageously bold claims about getting smarter and richer and sexier of the sort which virulent memes trying to propagate would have to make.

I don't think this creepiness detector operates on the conscious level, any more than the chance-of-catching-a-disease detector that tells us people with open sores all over their body are disgusting operates on a conscious level. We don't stop considering the open sores disgusting if we learn they're not really contagious, and we don't stop considering overblown self-improvement claims from an actively recruiting community to be a particularly virulent memetic vector even if we don't consciously believe that's what's going on.

(I'm still agnostic on that point. I'm sure no one intended this to be a meme optimized for self-propagation through outlandish promises, but it's hard to tell if it's started mutating that way.)

Comment author: lukeprog 03 July 2011 03:25:44AM 1 point [-]

I'm still working on posts for the Science of Winning at Life series and also a 'Summarizing the Sequences' sequence. I would love to put together meetup materials, too, but I doubt I'll be getting to that anytime soon. I've got a bunch of academic stuff to research and write.

Comment author: Yvain 03 July 2011 04:39:13AM 1 point [-]

Tell me more about your summarizing the sequences sequence (by PM if need be). I was thinking of trying something along those lines myself once my current projects are done, but I don't want to duplicate effort. I'm happy to leave it to you or we could see if we could split the effort (though I can't commit to anything until I learn my schedule for the summer/next year).

Comment author: Yvain 02 July 2011 09:20:08PM *  9 points [-]

Good post, and I agree with your conclusion, but I'd try a different tack to get there - one that's probably the same argument encoded differently.

Instead of talking about "desires for the well-being of others" vs. "self-interested desires", I'd throw out the whole desire language and use the language of behaviorism here. Seeing others in pain is negatively reinforcing. Helping others is positively reinforcing. Therefore we help others.

Instead of the question "what desire is at the root of your helping behavior?" this suggests the question "why is helping others reinforcing?" to which the answer is probably evolutionary and has nothing to do with the character of the person in question.

If you then ask the altruist what desires motivated their action, they'll make something up, the same way people usually make up their reasons for stuff.

Also, am I understanding the experiment table right in saying that when subjects felt high empathy for Elaine, they were more willing to stay when escaping was easy than when it was difficult? Any ideas why that might be?

Comment author: Yvain 01 July 2011 10:05:37AM 7 points [-]

Wow, thank you for the nice words.

I feel a predictable desire to reciprocate, but cataloging your output is starting to look like a Herculean task :)

Also, I can think of at least five of those posts that I no longer endorse and think are embarrassingly wrong, but I think it's more interesting if I don't say which ones.

Comment author: Yvain 27 June 2011 06:15:54AM 0 points [-]

You still want to carpool to LA on Tuesday night?

Comment author: KPier 26 June 2011 12:43:34AM 3 points [-]

I am not signed up for cryonics.

1) I think I can save more lives by being an organ donor. 2) I can't afford it, even with life insurance. 3) If there is a Singularity, I expect it will happen before I die anyway.

I can't actually sign up until I'm 18 even if all these are refuted, but I will precommit to signing up when I'm old enough.

Comment author: Yvain 26 June 2011 07:53:07PM *  2 points [-]

If number one is part of your true rejection, look to see if there are head-only cryonics available in your area.

Comment author: MatthewBaker 22 June 2011 03:29:46PM 0 points [-]

Good turnout :)

Comment author: Yvain 23 June 2011 05:33:03PM *  1 point [-]
Comment author: Laoch 28 May 2011 08:32:54AM 0 points [-]

I'm not sure if the Dublin meet is going ahead. I haven't heard any updates, have you?

Comment author: Yvain 28 May 2011 11:02:09PM 1 point [-]

Sorry I didn't see this before. Yes, it's going ahead. I'm in Dublin right now and look forward to seeing you tomorrow.

Comment author: Barry_Cotter 21 May 2011 01:56:16PM 3 points [-]

I'm coming. Have set things in my external memory to ensure I come. Currently living in Limerick.

Comment author: Yvain 21 May 2011 04:07:39PM *  0 points [-]

Consider co-ordinating a ride with Laoch; he lives in Limerick too. Also, let's make sure there are actually some Dubliners coming and it's not just Munster people, or we'll change the location to somewhere more convenient to ourselves.

If you have work on Monday, would an earlier time or Saturday be easier for you?

Irish Less Wrong meetup Sunday 29 May?

4 Yvain 21 May 2011 12:24PM

Louie's LW-visits-per-city statistics put Dublin at number 18 in the world, so there have got to be some Irish LWers lurking about. I'd like to have a meetup at...let's say 2 PM on Sunday 29 May at the Starbucks at 20 College Green near Trinity College in Dublin.

I am from Cork, and I'd like to convince some people from Limerick and further afield to come out, but it's a long trip to make without any reason to believe there will be more people than just myself there. So if you're interested in coming, please either post here or send me an email at scott [period] siskind [at-symbol] gmail [period] com.

Discussion will probably be about organizing future meetups, availability of rationalist/transhumanist networks and activities in Ireland, and other rationality and random chit-chat.

I'm not too attached to the date, time, or location, so we can change it if it conflicts with anything.

Comment author: Yvain 19 May 2011 07:30:30PM 6 points [-]

Are you saying this is one potential source of suffering, or are you defining suffering as those things which fit this pattern?

Comment author: [deleted] 18 May 2011 06:50:13AM 37 points [-]

This offers food for thought about various anti-aging strategies. For example, given the superexponential growth in mortality, if we had a magic medical treatment that could cut your mortality risk in half but didn't affect the growth of said risk, then that would buy you very little late in life, but might extend life by decades if administered at a very young age.

This isn't an anti-aging strategy, but it is an anti-death strategy: low-dose aspirin. As explained in this New York Times article on December 6, 2010, "researchers examined the cancer death rates of 25,570 patients who had participated in eight different randomized controlled trials of aspirin that ended up to 20 years earlier".

Eight. Different. Randomized. Controlled. Trials. Twenty-five thousand people.

They found (read the article) that low-dose aspirin dramatically decreased the risk of death from solid tumor cancers. Again, this ("risk of death") is the gold standard - many studies measure outcomes indirectly (e.g. tumor size, cholesterol level, etc.) which leads to unpleasant surprises (X shrinks tumors but doesn't keep people alive, Y lowers cholesterol levels but doesn't keep people alive, etc.). Best of all is this behavior: "the participants in the longest lasting trials had the most drastic reductions in cancer death years later."

Not mentioned in the article is the fact that aspirin is an ancient drug, in use for over a century with side effects that, while they certainly exist, are very well understood. This isn't like the people taking "life-extension regimens" or "nootropic stacks", who are, as far as I'm concerned, finding innovative ways to poison themselves.

Yet the article went on to say this:

But even as some experts hailed the new study as a breakthrough, others urged caution, warning people not to start a regimen of aspirin without first consulting a doctor about the potential risks, including gastrointestinal bleeding and bleeding in the brain (hemorrhagic strokes).

“Many people may wonder if they should start taking daily aspirin, but it would be premature to recommend people starting taking aspirin specifically to prevent cancer,” said Eric J. Jacobs, an epidemiologist with the American Cancer Society.

I'm a programmer, not a doctor - but after looking around, I concluded that the risks of GI bleeding were not guaranteed fatal, and the risks of hemorrhagic strokes were low in absolute terms. Also, aspirin is famously effective against ischemic strokes. According to Wikipedia: "Although aspirin also raises the risk of hemorrhagic stroke and other major bleeds by about twofold, these events are rare, and the balance of aspirin's effects is positive. Thus, in secondary prevention trials, aspirin reduced the overall mortality by about a tenth."

So unless aspirin's risks are far more grave than I've currently been led to believe, as far as I'm concerned, people saying "hey, even if you're not subject to aspirin's well-known contraindications, you shouldn't start low-dose aspirin just yet" are literally statistically killing people. Cancer is pretty lethal and we're not really good at fixing it yet, so when we find something that can really reduce the risk (and there aren't many - the only other ones I can think of are the magical substances known as not-smoking and avoiding-massive-doses-of-ionizing radiation), we should be all over that like cats on yarn.

I make damn sure to take my low-dose aspirin every day. I started it before reading this article on the advice of my doctor who thought my cholesterol was a little high - I'm almost 28, so it'll have many years in which to work its currently poorly understood magic.

That said, this reduces the risk of one common cause of death (two or three if you throw in heart attacks and ischemic strokes). There are lots of others out there. Even if you could avoid all of them (including the scariest one, Alzheimer's - it's insanely common, we have no fucking clue what causes it or how to stop it, and it annihilates your very self - even if cryonics is ultimately successful, advanced Alzheimer's is probably the true death), humans pretty clearly wear out with an upper bound of 120 years. Maybe caloric restriction can adjust that somewhat. But I think I'll sign up for cryonics sooner rather than later - I'm in favor of upgrading probability from "definitely boned" to "probably boned but maybe not".

Comment author: Yvain 19 May 2011 12:56:25PM *  23 points [-]

The meta-analysis you cite is moderately convincing, but only moderately. They had enough different analyses such that some would come out significant by pure chance. Aspirin was found to have an effect on 15-year-mortality significant only at the .05 level, and aspirin was found not to have a significant effect 20-year-mortality, so take it with a grain of salt. There was also some discussion in the literature about how it's meta-analyzing studies performed on people with cardiac risk factors but not bleed risk factors, and so the subjects may have been better candidates for aspirin than the general population.

The Wikipedia quote you give is referring to secondary prevention, which means "prevention of a disease happening again in someone who's already had the disease". Everyone agrees aspirin is useful for secondary prevention, but there are a lot of cases where something useful for secondary prevention isn't as good for primary. In primary prevention, aspirin doesn't get anywhere near a tenth reduction in mortality (although it does seem to have a lesser effect).

I would say right now there's enough evidence that people who enjoy self-experimentation are justified in trying low-dose aspirin and probably won't actively hurt themselves (assuming they check whether they're at special risk of bleeds first), but not enough evidence that doctors should be demonized for not telling everyone to do it.

Comment author: Yvain 18 May 2011 12:48:45AM 4 points [-]
In response to comment by Yvain on What we're losing
Comment author: lukeprog 16 May 2011 07:13:19PM *  12 points [-]

I agree that theoretical-sciency-mathy-insightful stuff is less common now than when Eliezer was writing posts regularly. I suspect this is largely because writing such posts is hard. Few people have that kind of knowledge, thinking ability, and writing skills, and the time to do the writing.

As someone who spends many hours writing posts only to have them nit-picked to death by almost everyone who bothers to comment, I appreciate your advice to "express small disagreements in 'IAWYC but' form."

As for your suggestion to downvote posts that "don't present non-obvious new material," I'm not sure what to think about that. My recent morality post probably contains only material that is obvious to someone as thoroughly familiar with LW material as yourself or Phil Goetz or Will Newsome or Vladimir Nesov or many others, but on the other hand a great many LWers are not quite that familiar, or else haven't taken the time to apply earlier lessons to a topic like morality (and were thus confused when Eliezer skipped past these basics and jumped right into 'Empathic Metaethics' in his own metaethics sequence).

In response to comment by lukeprog on What we're losing
Comment author: Yvain 18 May 2011 12:21:48AM *  0 points [-]

I enjoyed your morality post, as I do most of your posts, and certainly wouldn't accuse it of not presenting non-obvious new material.

Comment author: wedrifid 16 May 2011 12:55:36PM 0 points [-]

Did Yvain end up writing said post? That theory is approximately how I model self-esteem and it serves me well but I haven't seen what a formal theory on the subject looks like.

Comment author: Yvain 16 May 2011 12:59:15PM *  4 points [-]

http://lesswrong.com/lw/1kr/that_other_kind_of_status/ involves that idea; for the formal theory, Google "sociometer".

In response to comment by Yvain on What we're losing
Comment author: jsalvatier 15 May 2011 08:03:43PM 2 points [-]

Perhaps it would be easier and/or more constructive to comment 'I don't disagree with anything here, but I don't think this is valuable'?

Comment author: Yvain 15 May 2011 09:04:36PM 3 points [-]

Perhaps, but I expect far fewer people would do so: it's less anonymous and more likely to cause confrontations/bad feelings.

In response to What we're losing
Comment author: Eugine_Nier 15 May 2011 04:11:39AM 16 points [-]

I'm somewhat puzzled by your terminology since the topics you call "meta-rationality":

about how to be rational, how to avoid akrasia, and so on.

strike me as much more practical and applied then the ones you call "applied rationality":

philosophy, value, and possible futures

which strike me as much more meta.

Going by the list of topics you're complaining about, it appears that you are the one who "would rather talk about rationality than use it."

Comment author: Yvain 15 May 2011 04:42:05PM *  22 points [-]

Phil's terminology is probably the way I would have worded the same.

Posts that talk about things like "how do we use the anthropic principle", "what is morality", "what decision theory makes sense", "what is a mysterious answer to a mysterious question", etc. all seem object-level...

...whereas there's another class of posts that always uses the word "rationality" - ie "how can we be more rational in our lives", "how can we promote rationality", "am I a good enough rationalist if..." "who is/isn't a rationalist" et cetera, and these seem properly termed meta-level because they involve being rational about rationality.

I have a feeling the latter class of posts would benefit if they tried to taboo "rationality".

In response to What we're losing
Comment author: Yvain 15 May 2011 04:35:13PM *  42 points [-]

Agreed.

One person at the Paris meetup made the really interesting and AFAICT accurate observation that the more prominent a Less Wrong post was, the less likely it was to be high quality - ie comments are better than Discussion posts are better than Main (with several obvious and honorable exceptions).

I think maybe it has to do with the knowledge that anything displayed prominently is going to have a bunch of really really smart people swarming all over it and critiquing it and making sure you get very embarrassed if any of it is wrong. People avoid posting things they're not sure about, and so the things that get main-ed tend to be restatements of things that create pleasant feelings in everyone reading them without rocking any conceivable boat, and the sort of overly meta- topics you're talking about lend themselves to those restatements - for example "We should all be more willing to try new things!" or "Let's try to be more alert for biases in our everyday life!"

Potential cures include greater willingness to upvote posts that are interesting but non-perfect, greater willingness to express small disagreements in "IAWYC but" form, and greater willingness to downvote posts that are applause lights or don't present non-obvious new material. I'm starting to do this, but hitting that downvote button when there's nothing objectively false or stupid about a post is hard.

Comment author: Rain 14 May 2011 11:02:06PM *  39 points [-]

For every non-duplicate comment replying to this one praising me for my right action, I will donate $10 to SIAI, up to a cap of $1010, with the count ending on 1 June 2011. Also accepting private messages.

Edit: The cap was met on 30 May. Donation of $1010 made.

Comment author: Yvain 15 May 2011 12:18:35AM 3 points [-]

This is an excellent action! Commendations and praise be to you!

Comment author: Yvain 14 May 2011 06:44:53PM *  4 points [-]

Exciting. I'll make it to least a few of the Irvine ones in June and hopefully a few more in August.

Comment author: PlaidX 06 May 2011 04:00:42PM *  0 points [-]

Aha, I had a nagging feeling there might be something like that going on.

Any idea what the involuntary spasms are about? I did another hour of sitting, and while I didn't have the tingling and such this time, the spasm came back as strong as ever. In fact, I'm inclined to discontinue things until I can figure out what the deal is with them.

Even laying down, breathing calmly, I'm just twitchy as hell. It stops as soon as I stop meditating.

EDIT: Here's something from wikipedia.

Cortical reflex myoclonus is thought to be a type of epilepsy that originates in the cerebral cortex - the outer layer, or "gray matter," of the brain, responsible for much of the information processing that takes place in the brain. In this type of myoclonus, jerks usually involve only a few muscles in one part of the body, but jerks involving many muscles also may occur. Cortical reflex myoclonus can be intensified when patients attempt to move in a certain way or perceive a particular sensation. [italics mine]

This seems like a pretty bizarre explanation, but I have yet to uncover anything better. Wait, that's probably intended to be read "when patients perceive", not "when patients attempt to perceive".

Comment author: Yvain 06 May 2011 04:53:17PM *  2 points [-]

You're unlikely to have epilepsy. That's serious stuff.

Meditators commonly report twitches (here is an annoying New Age page about them, because it was the first one I could find).

I don't have any hard knowledge about them but my wild guess is that they're similar to hypnic jerks, basically your brain noticing it hasn't heard from your body lately and pinging it to make sure it's still there. The more serious twitches that get linked to kriyas are probably something more exotic, but what you're talking about doesn't sound like that.

If you're tired, sleep better and they might go away. If not, see if you can make meditation less of a relaxing brink-of-sleep-inducing experience by some of the tips David mentioned above.

The exceptionally large amount of twitching you're having now could also be linked to the previous hyperventilation. Note the part of the Wikipedia page that says alkalosis can cause "tetany" - that's involuntary muscle contraction. See if it goes away after a while breathing normally. Note that breathing normally during meditation is hard, at least for me.

Comment author: PlaidX 06 May 2011 02:29:52PM *  5 points [-]

WOAH, holy crap. Ok, I'm doing a retreat (in my own house, by myself) and i'm only four and a half hours in, but i'm breaking retreat protocol and going on the computer because I have to tell you guys how unexpected what's happened so far is. Woo, ok, sensations subsiding, getting feeling back in my fingers.

I've been meditating for about six months now, starting at 20 minutes a day and gradually moving up to an hour and a half, with no discernible effect other than my butt getting sore. When daniel posted these articles, I was getting so demoralized with my complete lack of results that I was looking into maybe kickstarting things with LSD or something.

Since he really sounds like he knows what he's talking about, and presents the information in a manner that's refreshingly candid and sassy, I decided to at least TRY doing a retreat, however painful it was. I expected maybe kinda sorta to get palpable results if I could stick with it for a few days.

Instead, here's what's happened so far. I actually started last night, but immediately went to bed, and just tried to be mindful in the moments I was awake, haphazardly. Got up, sat for fifteen minutes, nothing, walked for an hour, still absolutely nothing, sat for an hour, and here something at least worth writing down happened, which is that near the end I started getting a bit twitchy.

I'm prone to the occasional twitch or spasm in my daily life, but no more than a few per day. But I was starting to get them a couple of times a minute. I noted them, and eventually time was up, and I felt a little smoother or more clearheaded as I (slowly, mindfully) went downstairs and made myself a protein shake for breakfast, but certainly nothing out of the ordinary.

Then, more walking, still nothing, more sitting, and soon the twitching came back. Then it subsided for a while. Then it came back. Then it got worse. Then it kept getting worse and worse, more and more twitching, jerking, spasming. It seemed to be related to the noting process, if I let my mind wander or concentrated on the breath without noting "in in in in in out out out out out", the jerking subsided.

Eventually it started to get kind of painful, some of the random unpredictable jerks hurt my neck and whatnot, so experimenting I found that if I breathed faster (and consequently noted faster, "ininininin outoutoutoutout"), the twitching would come even faster, but in smaller, tighter, more controlled jerks, and in fact I was tensing up like crazy. I noticed a kind of buzzy vibration in my teeth and considered "Is that the vibrations he's talking about? No, i think that's just something irrelevant. That happens to people now and then". In retrospect, I'm not sure it does. It wasn't a REMARKABLE sensation, but I don't really recall having my teeth buzz normally.

But, the buzzing sensation did not stop in my teeth. Soon I was feeling it in my hands as well, and then spreading from my neck to my face, and eventually all over. Meanwhile my breath had become quite labored and audible and irregular and I'm sure had anyone else been trying to meditate in the same room they would've been greatly irritated.

My body felt very much the way your foot does when it falls asleep, though without any accompanying loss of sensation. Also I was getting a headache. And most weirdly of all, my fingers were bending back of their own accord. I found that regardless of how hard I tried, I could not hold my hands in the proper posture, they were all screwed up. Similarly my shoulders were hunched and my face was somewhat puckered, I think.

The vibrating / tingling / asleep body part feeling was mildly unpleasant. I've smoked salvia once or twice (actually it's the only drug i've done),and gives a certain prickly sensation all over. This somewhat resembled that as well.

When I had started out this retreat, my feeling was "if i can just get my foot in the door, i'm sure I can make it to the end", but not really expecting to get my foot in the door. Now that I have, I can see it's going to be just as difficult taking the next step as well. This was not an easy thing to sit with for even part of an hour, and in fact I gave up and broke my meditation a few seconds before the bell.

The world looked considerably... brighter? More detailed but in its normal textures? definitely different, in a positive way. The same way it looked after the first hour, which I dismissed, except much more so.

I'm surprised this happened so fast, and I'm surprised at how physical it was. What I've experienced so far did not feel in any way enlightening, but nor was it my imagination playing tricks on me. This is serious business.

Anyway, I'd better get back to my schedule. After typing all this, I'll probably have to start all over.

Comment author: Yvain 06 May 2011 03:00:44PM *  10 points [-]

I believe you are describing paraesthesiae from hyperventilation-induced respiratory alkalosis - ie you're breathing in too much oxygen too quickly and breathing out too much CO2 too quickly, it's turning your blood alkaline, and that's screwing with your nervous system.

It's not uncommon to mistake this for a spiritual result of breathing-related practices - I used to do so myself - but it isn't, it's not healthy, and you should try to avoid it by breathing at a more measured rate.

In response to comment by Yvain on Consequentialism FAQ
Comment author: Vladimir_M 05 May 2011 08:40:24PM *  16 points [-]

Each of these issues could be the subject of a separate lengthy discussion, but I'll try to address them as succinctly as possible:

  1. Re: phlogiston. Yes, Eliezer's account is inaccurate, though it seems like you have inadvertently made even more out of it. Generally, one recurring problem in the writings of EY (and various other LW contributors) is that they're often too quick to proclaim various beliefs and actions as silly and irrational, without adequate fact-checking and analysis.

  2. Re: interpersonal utility aggregation/comparison. I don't think you can handwave this away -- it's a fundamental issue on which everything hinges. For comparison, imagine someone saying that your consequentialism is wrong because it's contrary to God's commands, and when you ask how we know that God exists and what his commands are, they handwave it by saying that theologians have some ideas on how to answer these questions. In fact, your appeal to authority is worse in an important sense, since people are well aware that theologians are in disagreement on these issues and have nothing like definite unbiased answers backed by evidence, whereas your answer will leave many people thinking falsely that it's a well-understood issue where experts can provide adequate answers.

  3. Re: economists and statisticians. Yes, nowadays it's hard to deny that central planning was a disaster after it crumbled spectacularly everywhere, but read what they were saying before that. Academics are just humans, and if an ideology says that the world is a chaotic inefficient mess and experts like them should be put in charge instead, well, it will be hard for them to resist its pull. Nowadays this folly is finally buried, but a myriad other ones along similar lines are actively being pursued, whose only redeeming value is that they are not as destructive in the short to medium run. (They still make the world uglier and more dysfunctional, and life more joyless and burdensome, in countless ways.) Generally, the idea that you can put experts in charge and expect that they their standards of expertise won't be superseded by considerations of power and status is naively utopian.

  4. Re: procedures in place for violating heuristics. My problem is not with the lack of elegant philosophical rules. On the contrary, my objections are purely practical. The world is complicated and the law of unintended consequences is merciless and unforgiving. What's more, humans are scarily good at coming up with seemingly airtight arguments that are in fact pure rationalizations or expressions of intellectual vanity. So, yes, the heuristics must be violated sometimes when the stakes are high enough, but given these realistic limitations, I think you're way overestimating our ability to identify such situations reliably and the prudence of doing so when the stakes are less than enormous.

  5. Re: Section 7. Basically, you don't take the least convenient possible world into account. In this case, the LCPW is considering the most awful thing imaginable, assuming that enough people assign it positive enough value that the scales tip in their favor, and then giving a clear answer whether you bite the bullet. Anything less is skirting around the real problem.

  6. Re: welfare of some more than others. I'm confused by your position: are you actually biting the bullet that caring about some people more than others is immoral? I don't understand why you think it's weird to ask such a question, since utility maximization is at least prima facie in conflict with both egoism and any sort of preferential altruism, both of which are fundamental to human nature, so it's unclear how you can resolve this essential problem. In any case, this issue is important and fundamental enough that it definitely should be addressed in your FAQ.

  7. Re: game theory and the thought process. The trouble is that consequentialism, or at least your approach to it, encourages thought processes leading to reckless action based on seemingly sophisticated and logical, but in reality sorely inadequate models and arguments. For example, the idea that you can assess the real-world issue of mass immigration with spherical-cow models like the one to which you link approvingly is every bit as delusional as the idea -- formerly as popular among economists as models like this one are nowadays -- that you can use their sophisticated models to plan the economy centrally with results far superior to those nasty and messy markets.

General summary: I think your FAQ should at the very least include some discussion of (2) and (6), since these are absolutely fundamental problems. Also, I think you should research more thoroughly the concrete examples you use. If you've taken the time to write this FAQ, surely you don't want people dismissing it because parts of it are inaccurate, even if this isn't relevant to the main point you're making.

Regarding the other issues, most of them revolve around the general issues of practical applicability of consequentialist ideas, the law of unintended consequences (of which game-theoretic complications are just one special case), the reliability of experts when they are in positions where their ideas matter in terms of power, status, and wealth, etc. However you choose to deal with them, I think that even in the most basic discussion of this topic, they deserve more concern than your present FAQ gives them.

Comment author: Yvain 05 May 2011 10:00:49PM *  3 points [-]

Okay, thank you.

I will replace the phlogiston section with something else, maybe along the lines of the example of a medicine putting someone to sleep because it has a "dormitive potency".

I agree with you that there are lots of complex and messy calculations that stand between consequentialism and correct results, and that at best these are difficult and at worst they are not humanly feasible. However, this idea seems to me fundamentally consequentialist - to make this objection, one starts by assuming consequentialist principles, but then saying they can't be put into action and so we should retreat from pure consequentialism on consequentialist grounds. The target audience of this FAQ is people who are not even at this level yet - people who don't even understand that you need to argue against certain "consequentialist" ideas on consequentialist grounds, but rather that they can be dismissed by definition because consequences don't matter. Someone who accepts consequentialism on a base level but then retreats from it on a higher level is already better informed than the people I am aiming this FAQ at. I will make this clearer.

This gets into the political side of things as well. I still don't understand why you think consequentialism implies or even suggests centralized economic planning when we both agree centralized economic planning would have bad consequences. Certain decisions have to be made, and making them on consequentialist grounds will produce the best results - even if those consequentialist grounds are "never give the government the power to make these decisions because they will screw them up and that will have bad consequences". I continue to think prediction markets allow something slightly more interesting than that, and I think if you disagree we can resolve that disagreement only on consequentialist grounds - eg would a government that tried to intervene where prediction markets recommended intervention create better consequences than one that didn't. Nevertheless, I'll probably end up deleting a lot of this section since it seemed to give everyone an impression I don't endorse.

Hopefully the changes I listed in my other comment on this thread should help with some of your other worries.

Comment author: DavidM 04 May 2011 11:38:06PM *  6 points [-]

The earliest contemporary tradition which emphasizes meditation in a style like this that I know of goes back to Mahasi Sayadaw, a Burmese Theravada monk, and then to whoever taught him. Mahasi Sayadaw's impact on Theravada Buddhism has been very large, though his tradition is certainly not the only one in contemporary Theravada Buddhism.

"Vibrations" is a term you won't see in any Buddhist literature. Where I write "pay attention to vibrations," Buddhists would write "pay attention to the impermanence characteristic of phenomena." In Buddhism, "impermanence" (Pali: anicca) is one of the three characteristics of everything that exists. "Vibrations" is a term that suits the positivist in me better. (Note: I didn't invent the term. It's common in contemporary secular communities interested in enlightenment.)

What you're describing doesn't sound like vibrations; there are other [for lack of a better word] "pulsatory" phenomena in experience. But you can make an educated guess under the following assumption: if you can see them that easily, you should be able to see them in lots of places. So, look at part of your visual field and see if it has any property that you would describe as 'vibratory.' Or touch your skin and see if the sensation has any vibratory property.

I don't know what's up with the particular numbers. There may be some relationship between the frequencies of vibrations and the frequencies of brain waves, but I wouldn't know. Vibrations typically present at frequencies around there (5,7, 10, whatever), and it is true that certain frequencies are more predominant in certain stages. It would be interesting to know whether there is an interesting association between the stages, their typical cognitive / emotional / attentional / perceptual manifestations, and brainwave activity.

EDITED FOR CLARITY: I didn't see that you wrote alpha waves. Vibrations often present slower and faster than alpha waves. I've seen 3-4hz often enough. Stage two often has really high frequency vibrations (20hz+) if your concentration is strong or you're deep enough into it.

I've been through the stages I've described, yes.

Comment author: Yvain 05 May 2011 12:33:38PM 0 points [-]

Thank you. I'd heard the "impermanence of phenomena" phrase before but mistakenly thought it meant things like how all your material goods will one day break down and decay. I'll look up Sayadaw.

Comment author: Yvain 04 May 2011 11:12:46PM *  4 points [-]

Interesting. Thank you.

Can you explain a little about where this style of meditation comes from? I only know the bare basics of a few Buddhist styles, but I'd never heard before this emphasis on "vibrations" (I've only heard them mentioned a few times as some sort of metaphysical thing).

Trying it very briefly, I find myself noticing several vibration-like processes (saccadic eye movements, micro-level catches and inconsistencies in the regularity of breathing, pounding of blood in facial arteries). Should real vibrations be obvious and distinct from all these when you perceive them?

I also notice you give numbers like 7 hz and 10 hz which are similar to the frequency of alpha waves. Is this intentional?

Have you personally been through all these stages? Has anyone else on LW done so? I'm not going to ask for a controlled trial, but knowing more than one person has had this experience would be significant.

In response to Consequentialism FAQ
Comment author: Yvain 04 May 2011 03:09:48PM 5 points [-]

Okay, summary of things I've learned I should change from this feedback:

  1. Fix dead links (I think OpenOffice is muddling open quotes and close quotes again)

  2. Table of contents/navigation.

  3. Stress REALLY REALLY REALLY HARD that this is meant to be an introduction and that there's much more stuff like game theory and decision theory that is necessary for a full understanding of utilitarianism.

  4. Change phlogiston metaphor subsequent to response from Vladimir_M

  5. Remove reference to Eliezer in "warm fuzzies" section.

  6. Rephrase "We have procedures in place for violating heuristics" to mention that it's patchwork and we still don't have elegant rules for this sort of thing.

  7. In part about using utilitarianism to set policy, stress involvement of clever tools like prediction markets to lessen the immediate reaction that I'm obviously flaming mad.

  8. Rephrase gladiatorial games to be more clear.

  9. Possibly change title to "Rarely Asked Questions" or "Never Asked Questions" or just keep the abbreviation the same but expand it to "Formatted Answers and Questions"

  10. Remove reference to desire utilitarianism until I'm sure I understand it.

  11. Change 'most consequentialists give similar results' to 'most popular consequentialisms...'

  12. Change bit about axiology collapsing distinctions.

  13. Consequentialism chooses best state of world ---> consequentialism chooses better world-state of two actions.

  14. Add to part about fat man that although the example is correct insofar as it correctly teaches consequentialist reasoning, game/decision theory type considerations might change the actual correct answer.

In response to Consequentialism FAQ
Comment author: XiXiDu 27 April 2011 09:43:08AM 1 point [-]

Here is what David Pearce has to say about the FAQ (via Facebook):

Lucid. But the FAQ (and lesswrong) would be much stronger if it weren't shot through with anthropocentric bias...

Suggestion: replace "people" with "sentient beings".

For the rational consequentialist, the interests of an adult pig are (other things being equal) as important as a human toddler. Sperm whales are probably more sentient than human toddlers; chickens probably less.

Ethnocentric bias now seems obvious. If the FAQ said "white people" throughout rather than "people", then such bias would leap off the page - though it wouldn't to the Victorians.

Sadly, anthropocentric bias is largely invisible.

Comment author: Yvain 04 May 2011 03:08:21PM 0 points [-]

I can't even use nonstandard pronouns without it impeding readability, so I think I'm going to sacrifice precision and correctness for the sake of ease-of-understanding here.

In response to Consequentialism FAQ
Comment author: PlaidX 26 April 2011 02:40:58AM 39 points [-]

I like it, but stop using "ey". For god's sake, just use "they".

Comment author: Yvain 04 May 2011 03:02:02PM 2 points [-]

I agree that "ey" is annoying and distracting, but I feel like someone's got to be an early adopter or else it will never stop being annoying and distracting.

In response to Consequentialism FAQ
Comment author: MarkLee 28 April 2011 12:57:41AM 0 points [-]

Part One: Methodology: Why think that intuitions are reliable? What is reflective equilibrium, other than reflecting on our intuitions? If it is some process by which we balance first-order intuitions against general principles, why think this process is reliable? Metaethics: Realism vs. Error vs. Expressivism?

Part Two: 2.6 I don't see the collapse - an axiology may be paired with different moralities - e.g. a satisficing morality, or a maximizing morality. Maybe all that is meant by the collapse is that the right is a function of the good? Then 'collapse' is misleading.

Part Four: 4.2 Taking actions that make the world better is different from taking actions that make the world best. Consequentialism says that only consequences matter - a controversial claim that hasn't been addressed.
4.4 Makes a strawman of the deontologist. Deontologists differ from consequentialists in ways other than avoiding dirtying their hands / guilt. They may care about not using others as means, or distinctions like doing/allowing, killing/letting die, etc., which apply to some trolley cases, and (purportedly) justify not producing the best consequences. More argument is needed to show that this precludes morality from 'living in the world'.

Part Five: 5.4 Not obvious that different consequentialisms converge on most practical cases. Some desire pain. Some desire authenticity, achievement, relationships, etc. (no experience machine). Some desire not to be cheated on / have their wills disregarded / etc.

Part Seven: 7.3 Doesn't address the strongest form of the objection. A stronger form is: we know that certain acts or institutions are necessarily immoral (gladiatorial games, slavery); utilitarianism could (whether or not it does) require we promote these; therefore utilitarianism is false. I like the utility monster example of this. The response in 7.5 to the utility monster case is bullet-biting - this should be the response in 7.3. The response that utilitarianism probably won't tell us to promote these is inadequate. The mistake is remade by the three responses in 7.4 (prior to the appeal to ideal rather than actual preferences). 7.6 Similar problem here. The response quibbles with contingent facts, but the force of the objection is that vicious, repugnant, petty, stupid, etc., preferences have no less weight in principle, i.e. in virtue of their status as such. 7.7 Response misses the point. The objection is that it's hard to see how utilitarianism can accommodate the intuitive distinction between higher and lower pleasures. Sure, utilitarians have nothing against symphonies, but would a world with symphonies be best? (Would an FAI-generated world contain symphonies?) 7.9 Rather quick treatment of the demandingness objection. One relevant issue in the vicinity is that of agent-centered permissions - permissions to do less than the best (in consequentialist terms), e.g. to favor those with whom we have special relations. Many philosophers and folk alike believe in such permissions - utilitarianism has a counterintuitive result here.

Suggestions for further content: (1) How are we to conceive of 'better' consequences? Perhaps any of the answers given by the aforementioned systems would suffice - pleasure, preference satisfaction, ideal preference satisfaction. But I'm not convinced these are practically/pragmatically equivalent. For instance, there may be different best methods for investigating what produces the most pleasure vs. what would best satisfy our ideal preferences, and so different practical recommendations. (2) What's our axiology? Is it total utilitarian, egalitarian, prioritarian, maximizing, satisficing, etc.? How do the interests of animals, future time slices, and future individuals weigh against present human interests? A total utilitarian approach seems to be advocated, but that faces its own set of problems (repugnant conclusions, fanaticism, etc.).

Comment author: Yvain 04 May 2011 02:58:41PM 0 points [-]

P1: Intuitions being "reliable" requires that the point of intuitions be to correspond to something outside themselves. I'm not sure moral intuitions have this point.

P2: Point taken.

P4.2: I agree with taking actions that make the world better instead of best and will rephrase. I don't understand the point of your second sentence.

4.4: Concern about not using others as means, or doing/allowing distinctions, seem to me common-sensically not to be about states of the world. I'm not sure what further argument is possible let alone necessary. The discussion of guilt only says that's the only state-of-the-world-relevant difference.

5.4: Would you agree that most of the philosophically popular consequentialisms (act, rule, preference, etc.) usually converge?

7.3 and below: I don't think slavery and gladiators are necessarily wrong. I can imagine situations in which they would be okay (I've mentioned some for gladiators above) and I remain open to moral argument from people who want to convince me they're okay in our own world (although I don't expect this argument to succeed any more than I expect to be convinced that the sky is green).

If the belief that slavery is wrong is not an axiom, but instead derives from deeper moral principles that when formalized under reflective equilibrium give you consequentialism, then I think it's fair to say that consequentialism proves they are wrong, but that in a counterfactual world where consequentialism proved they were right, I would either have intuitions that they were right, or be willing to discard my intuition that they were wrong after considering the consequentialist arguments against it.

In response to Consequentialism FAQ
Comment author: Matt_Simpson 27 April 2011 05:49:33AM 5 points [-]

One problem with the FAQ: The standard metaethics around here, at least EY's metaethics, is not utilitarianism. Utilitarianism says maximize aggregate utility, with "aggregate" defined in some suitable way. EY's metaethics says maximize your own utility (with the caveat that you only have partial information of your utility function), and that all humans have sufficiently similar utility functions.

You get something pretty similar to utilitarianism from that last condition (if everyone has the same utility function and you're maximizing your own utility function, then you're also maximizing aggregate utility in many senses of the term "aggregate"). But note that the interpersonal comparison of utility problem vanishes: you're maximizing your own utility function. Maximization of the aggregate (under some methods of comparisons) is merely a consequence of this, nothing more. Also note that if we relax that last condition and let humans have differing utility function, there is no intrinsic problem for EY's theory. If someone has a legitimate preference for killing people, the utilitarian has to take that into account as positive utility (or add some ad hoc assumptions about which preferences matter). On EY's theory, sans that last condition, you only take into account someone's preference for murder if your utility function tells you to. You may value other humans satisfying their preferences, but that doesn't mean you have to value every single preference every single human has. You could, but it really just depends on what your utility function says.

Comment author: Yvain 04 May 2011 02:35:18PM 0 points [-]

I'm glad you brought that up, since it's something I've mentally been circling around but never heard verbalized clearly before.

Both the classical and the Yudkowskian system seem to run into some problems that the other avoids, and right now I'm classifying the difference as "too advanced to be relevant to the FAQ". Right now my own opinions are leaning toward believing that under reflective equilibrium my utility function should reference the aggregate utility function and possible be the same as it.

In response to Consequentialism FAQ
Comment author: Morendil 27 April 2011 12:06:03PM *  3 points [-]

I'm looking forward to live discussion of this topic at the Paris meetup. :)

Meanwhile, I've read through it, more closely. Much of it seems, not necessarily right, but at least unobjectionable - it raises few red flags. On the other hand, I don't think it makes me much the wiser about the advantages of consequentialism.

Paras 7.2 and 7.3 (the slavery and gladiator questions) left me with an odd impression. The "test" you propose in both cases is more or less the same as Rawls' Veil of Ignorance. So at that point I was wondering, if you apply Rawls' procedure to determine what is a preferable social contract, perhaps you're a Rawlsian more than you're a consequentialist. :) BTW, are you familiar with Rawls' objections to (classical) utilitarianism?

Para 8.2 comes across as terribly naive, and "politics has been reduced to math" in particular seems almost designed to cause people to dismiss you. (A nitpick: the links at the end of 8.2 are broken.)

One thing that makes the essay confusing for me is the absence of a clear distinction between the questions "how do I decide what to do next" and "what makes for a desirable set of agreements among a large number of people" - between evaluating the morality of individual actions and choosing a social contract.

Another thing that's left out is the issue of comparing or aggregating happiness, or "utility", across different people. The one place where you touch on it, your response to the "utility monster" argument, does not match my own understanding of how a "utility monster" might be a problem. As I understood it, a "utility monster" isn't someone who is to you as you are to an ant, but someone just like you. They just happen to insist that an ice cream makes them a thousand times happier than they make you, so in all cases where it must be decided which of you should get an ice cream they should always get it.

Your analogy with optical illusions is apt, and it gives a good guideline for evaluating a proposed system of morality: in what cases does the proposed system lead me to change my mind on something that I previously did or avoided doing because of a moral judgement.

Interestingly, though, you give more examples that have to do with the social contract (gun control, birth control, organ donation policy, public funding of art, discrimination, slavery, etc.) than you give examples that have to do with personal decisions (giving to charities, trolley problems).

My own positions are contractarian, much more than they are deontological or consequentialist. I'm generally truthful, not because it is "wrong" to lie or because I have a rule against it (for instance I'm OK with lying in the context of a game, say Diplomacy, where the usual social conventions are known to be suspended - though I'd be careful about hurting others' feelings through my play even in such a context). I don't trust myself to compute the complete consequences of lying vs. not lying in each case, and so a literal consequentialism isn't an option for me.

However, I would prefer to live in a world where people can be relied upon to tell the truth, and for that I am willing to sacrifice the dubious advantage of being able to put a fast one over on other people from time to time. It is "wrong" to lie in the sense that if you didn't know ahead of time what particular position you'd end up occupying in the world (e.g. a politician with power) but only knew some general facts about the world, you would find a contract that banned lying acceptable, and would be willing to let this contract sanction lying with penalties. (At the same time, and for the same reason, I also put some value on privacy: being able to lie by omission about some things.)

I find XiXiDu's remarks interesting. It seems to me that at present something like "might makes right" is descriptively true of us humans: we could describe a morality only in terms of agreements and generally reliable penalties for violating these agreements. "If you injure others, you can expect to be put in prison, because that's the way society is currently set up; so if you're rational, you'll curb your desires to hurt others because your expected utility for doing so is negative".

However this sort of description doesn't help in finding out what the social contract "should" be - it doesn't help us find what agreements we currently have that are wrong because they result from the moral equivalent of optical illusions "fooling us" into believing something that isn't the case.

It also doesn't help us in imagining what the social contract could be if we weren't the sort of beings we are: if the agreements we enter were binding for reasons other than fear of penalties. This is a current limitation of our cognitive architectures but not a necessary one.

(I find this a very exciting question, and at present the only place I've seen where it can even be discussed is LW: what kind of moral philosophy would apply to beings who can "change their own source code".)

EDIT: having read Vladimir_M's reply below, his comments capture much of what I wanted to say, only better.

Comment author: Yvain 04 May 2011 02:32:16PM 0 points [-]

I've responded to some of Vladmir's comments, but just a few things you touched on that he didn't:

Utility monsters: if a utility monster just means someone who gets the same amount of pleasure from an ice cream that I get from an orgasm, then it just doesn't seem that controversial to me that giving them an ice cream is as desirable as giving me an orgasm. Once we get to things like "their very experience is a million times stronger and more vivid than you could ever imagine" we're talking a completely different neurological makeup that can actually hold more qualia, which is where the ant comes in.

I don't see a philosophical distinction between the morality an individual should use and the morality a government should use (although there's a very big practical distinction since governments are single actors in their own territories and so can afford to ignore some game theoretic and decision theoretic principles that individuals have to take into account). The best state of the world is the best state of the world, no matter who's considering it.

I use mostly examples from government because moral dilemmas on the individual level are less common, less standardized, and less well-known.

Comment author: CronoDAS 26 April 2011 05:48:24AM *  3 points [-]

(5.31) Desire utilitarianism replaces preferences with desire. The differences are pretty technical and I don't understand all of them, but desire utilitarians sure seem to think their system is better.

Then I would suggest either doing the research or not mentioning it, since this is not critical to the concept of consequentialism. I'm not entirely clear on it either.

Desire utilitarianism doesn't replace preferences with desires, it replaces actions with desires. It's not a consequentialist system; it's actually a type of virtue ethics. When confronted with the "fat man" trolley problem, it concludes that there are good agents that would push the fat man and other good agents that wouldn't. You should probably avoid mentioning it.

Comment author: Yvain 04 May 2011 01:58:40PM *  0 points [-]

Thank you. That makes more sense than the last explanation of it I read.

In response to Consequentialism FAQ
Comment author: AlephNeil 26 April 2011 05:24:33AM 10 points [-]

I think your analysis of the 'gladiatorial objection' misses something:

I hope I'm not caricaturing you too much if I condense your rebuttal as follows: "People wouldn't really enjoy watching gladiators fight to the death. In fact, they'd be sickened and outraged. Therefore, utilitarianism does not endorse gladiatorial games after all."

But there's a problem here: If the negative reaction to gladiatorial games is itself partly due to analyzing those games in utilitarian terms then we have a feedback loop.

Games are outrageous --> decrease utility --> are outrageous --> etc.

But this could just as well be a 'virtuous circle':

Games are noble --> increase utility --> are noble --> etc.

If we started off with a society like that of ancient Rome, couldn't it be that the existence of gladiatorial games is just as 'stable' (with respect to the utilitarian calculus) as their non-existence in our own society?

Couldn't it be that we came to regard such bloodsports as being 'immoral' for independent, non-utilitarian reasons*? And then once this new moral zeitgeist became prevalent, utilitarians could come along and say "Aha! Far from being 'fun', just look at how much outrage the games would generate. If only our predecessors had been utilitarians, we could have had avoided all this ugly carnage."

(Perhaps you will bite the bullet here, and grant that there could be a society where gladiatorial games are 'good' by utilitarian standards. But then there doesn't seem to be much hope for a utilitarian justification of the idea that, insofar as we have outlawed bloodsports, we have 'progressed' to a better state of affairs.

Or perhaps you will say that bloodsports would always be judged 'bad' under ideal rational reflection (that is, they go against our CEV). I think this is a much stronger reply, but it's not clear that CEV actually makes sense (i.e. that the limit is well-defined).)

* Sadly my knowledge of history is too meagre to venture an account of how this actually happened.

I have many other objections to utilitarianism up my sleeve. To give the gist of a few of them:

  1. Utilitarian calculations are impossible in practice because the future cannot be predicted sufficiently far.
  2. Utilitarian calculations are impossible even if theory because outcomes are incommensurable. The indeterminacies concerning whether 'more people' are preferable to 'happier people', and how far a superbeing's happiness is 'worth more' than a human's, are special cases of this, but incommensurability is ubiquitous. (For instance, just try weighing up all of the effects of the decision to buy a car rather than use public transport. The idea is there is a Right Answer Out There seems to me an article of blind faith.)
  3. Utilitarianism holds 'terminal preferences' to be beyond reproach. It does not allow for the possibility that an entire self-contained society ought to change its system of preferences, no matter how 'brutal' and 'destructive' these preferences are. (The point about gladiatorial games is a special case of this). It denies that one can make an objective judgement as to whether a paperclipper is 'wrong' and/or 'stupid' to fill the universe with paperclips. Ultimately, might makes right in the struggle between humanity and clippy.
  4. Utilitarianism faces some awkward choices in how it values the lives of 'ordinary people' (people who live reasonably happy lives but do not make lasting 'achievements' e.g. progress in science). If their value is positive then apparently it would be better to fill the universe with them than not, which seems absurd. How is it worthwhile or noble to try to explore the entire 'soap opera of Babel'? Isn't it just a stationary stochastic process? Haven't you seen it all once you've seen the first few billion episodes? But if their value is zero (resp. negative) then it seems that nuking an entire planet full of 'ordinary people', assuming it's not the only such planet, is morally neutral (resp. desirable). The only way of resolving the contradiction of human life being both incredibly precious and utterly worthless is to deny that the premise that we need to assign it some value in order to decide how to act.
Comment author: Yvain 04 May 2011 01:54:35PM *  1 point [-]

With the gladiatorial games, you seem to have focused on what I intended to be a peripheral point (I'll rephrase it later so this is clearer).

The main point is that forcing people to become gladiators against their will requires a system that would almost certainly lower utility (really you'd have to have an institution of slavery or a caste system; any other way and people would revolt against the policy since they would expect a possibility of being to be gladiators themselves).

Allowing people who want to, to become gladiators risks the same moral hazards brought up during debates on prostitution - ie maybe they're just doing it because they're too poor or disturbed to have another alternative, and maybe the existence of this option might prevent people from creating a structure in which they do have another alternative. I'm split on the prostitution debate myself, but in a society where people weren't outraged by gladiatorial games, I would be willing to bite the bullet of saying the gladiator question should be resolved the same way as the prostitute question.

In a utopian society where no one was poor or disturbed, and where people weren't outraged by gladiatorial games, I would be willing to allow people to become gladiators.

(in our current society, I'm not even sure whether American football is morally okay)

In response to Consequentialism FAQ
Comment author: Vladimir_M 27 April 2011 01:03:19AM *  40 points [-]

OK, I've read the whole FAQ. Clearly, a really detailed critique would have to be given at similar length. Therefore, here is just a sketch of the problems I see with your exposition.

For start, you use several invalid examples, or at least controversial examples that you incorrectly present as clear-cut. For example, the phlogiston theory was nothing like the silly strawman you present. It was a falsifiable scientific theory that was abandoned because it was eventually falsified (when it was discovered that burning stuff adds mass due to oxidation, rather than losing mass due to escaped phlogiston). It was certainly a reductionist theory -- it attempted to reduce fire (which itself has different manifestations) and the human and animal metabolism to the same underlying physical process. (Google "Becher-Stahl theory".) Or, at another place, you present the issue of "opposing condoms" as a clear-cut case of "a horrendous decision" from a consequentialist perspective -- although in reality the question is far less clear.

Otherwise, up to Section 4, your argumentation is passable. But then it goes completely off the rails. I'll list just a few main issues:

  • In the discussion of the trolley problem, you present a miserable caricature of the "don't push" arguments. The real reason why pushing the fat main is problematic requires delving into a broader game-theoretic analysis that establishes the Schelling points that hold in interactions between people, including those gravest ones that define unprovoked deadly assault. The reason why any sort of organized society is possible is that you can trust that other people will always respect these Schelling points without regards to any cost-benefit calculations, except perhaps when the alternative to violating them is by orders of magnitude more awful than in the trolley examples. (I have compressed an essay's worth of arguments into a few sentences, but I hope the main point is clear.)

  • In Section 5, you don't even mention the key problem of how utilities are supposed to be compared and aggregated interpersonally. If you cannot address this issue convincingly, the whole edifice crumbles.

  • In Section 6, at first it seems like you get the important point that even if we agree on some aggregate welfare maximization, we have no hope of getting any practical guidelines for action beyond quasi-deontologist heuristics. But they you boldly declare that "we do have procedures in place for breaking the heuristic when we need to." No, we don't. You may think we have them, but what we actually have are either somewhat more finely tuned heuristics that aren't captured by simple first-order formulations (which is good), or rationalizations and other nonsensical arguments couched in terms of a plausible-sounding consequentialist analysis (which is often a recipe for disaster). The law of unintended consequences often bites even in seemingly clear-cut "what could possibly go wrong?" situations.

  • Along similar lines, you note that in any conflict all parties are quick to point out that their natural rights are at stake. Well, guess what. If they just have smart enough advocates, they can also all come up with different consequentialist analyses whose implications favor their interests. Different ways of interpersonal utility comparison are often themselves enough to tilt the scales as you like. Further, these analyses will all by necessity be based on spherical-cow models of the real world, which you can usually engineer to get pretty much any implication you like.

  • Section 7 is rather incoherent. You jump from one case study to another arguing that even when it seems like consequentialism might imply something revolting, that's not really so. Well, if you're ready to bite awful consequentialist bullets like Robin Hanson does, then be explicit about it. Otherwise, clarify where exactly you draw the lines.

  • Since we're already at biting bullets, your FAQ fails to address another crucial issue: it is normal for humans to value the welfare of some people more than others. You clearly value your own welfare and the welfare of your family and friends more than strangers (and even for strangers there are normally multiple circles of diminishing caring). How to reconcile this with global maximization of aggregate utility? Or do you bite the bullet that it's immoral to care about one's own family and friends more than strangers?

  • Question 7.6 is the only one where you give even a passing nod to game-theoretical issues. Considering their fundamental importance in the human social order and all human interactions, and their complex and often counter-intuitive nature, this fact by itself means that most of your discussion is likely to be remote from reality. This is another aspect of the law of unintended consequences that you nonchalantly ignore.

  • Finally, your idea that it is possible to employ economists and statisticians and get accurate and objective consequentialist analysis to guide public policy is altogether utopian. If such things were possible, economic central planning would be a path to prosperity, not the disaster that it is. (That particular consequentialist folly was finally abandoned in the mainstream after it had produced utter disaster in a sizable part of the world, but many currently fashionable ideas about "scientific" management of government and society suffer from similar delusions.)

Comment author: Yvain 04 May 2011 01:46:21PM *  3 points [-]

Phlogiston: my only knowledge of the theory is Eliezer's posts on it. Do Eliezer's posts make the same mistake, or am I misunderstanding those posts?

Trolley-problem: Agreed about Schelling points of interactions between people. What I am trying to do is not make a case for pushing people in hypothetical trolley problems, but to show that certain arguments against doing so are wrong. I think I returned to some of the complicating factors later on, although I didn't go quite so deep as to mention Schelling points by name. I'll look through it again and make sure I've covered that to at least the low level that would be expected in an introductory argument like this.

Aggregating interpersonal utilities: Admitted that I handwave this away by saying "Economists have some ideas on how to do this". The FAQ was never meant to get technical, only provide an introduction to the subject. Because it is already 25 pages long I don't want to go that deep, although I should definitely make it much clearer that these topics exist.

Procedures in place for violating heuristics: By this I mean that we have laws that sometimes supervene certain rights. For one example, even though we have a right to free speech, we also have a law against hate speech. Even though we have a right to property, we also have laws of eminent domain when one piece of property is blocking construction of a railway or something. Would it be proper to rephrase your objection as "We don't have a single elegant philosophical rule for deciding when it is or isn't okay to violate heuristics"?

Parties pointing out natural rights are at stake: In a deontological system, these conflicts are not solveable even in principle: we simply don't know how to decide between two different rights and the only hope is to refer it to politicians or the electorate or philosophers. In a consequentialist system it's certainly possible to disagree, and clever arguers can come up with models in their favor, but it's possible to develop mathematical and scientific tools for solving the problem (for example, prediction markets would solve half of this and serious experimental philosophy could make a dent on the other half). And there are certain problems which are totally opaque to rights-based arguments which you couldn't even begin to argue on consequentialist grounds (eg opt-out organ example given later)

Section 7: I don't really understand your criticism. Yes, it's jumping from place to place. I'm answering random objections that people tend to bring up. Do you think I'm straw-manning or missing the important objections? The Nazi and slavery objections at your link seem very much like the racism and slavery objections addressed on the FAQ, and the Hannibal-the-baby-eater objection only seems relevant if one confuses money with utility.

Welfare of some more than others: I admit that I have these preferences, but I don't think they're moral preferences. I might choose to save my mother rather than two strangers, I just would be doing it for reasons other than morality. This strikes me as a really weird objection - is there some large group of people who say that nepotism is the moral thing to do?

Game theoretic issues: Agreed that these are important. This is meant to be an introductory FAQ to prime some intuitions, not a complete description of all human behavior. Given that game theory usually means that consequentialism is more likely to give the intuitively correct answer to moral dilemmas, I don't feel like I'm being dishonest or cherry-picking by excluding most mentions of it. (game theory is against consequentialism only if you mistake consequentialism for certain consequentialism-signaling actions, like pushing people in front of trolleys or assassinating Hitler, rather than considering it as the thought process generating these actions. Learn the thought process first, then master the caveats)

Regarding economists and statisticians: The widespread consensus of economists and statisticians is that economic central planning doesn't work. I would expect something like prediction markets to be not only be able to guide certain policies, but to be able to accurately predict where to use and where not to use prediction markets.

General response to your comments: Mostly right but too deep for the level at which this FAQ is intended. I will try to revise the FAQ to emphasize that the FAQ is intended only to teach consequentialist thought processes, and that these must then be modified by knowledge of things like game theory.

Consequentialism FAQ

20 Yvain 26 April 2011 01:45AM

There are a lot of explanations of consequentialism and utilitarianism out there, but not a lot of persuasive essays trying to convert people. I would like to fill that gap with a pro-consequentialist FAQ. The target audience is people who are intelligent but may not have a strong philosophy background or have thought about this matter too much before (ie it's not intended to solve every single problem or be up to the usual standards of discussion on LW).

I have a draft up at http://www.raikoth.net/consequentialism.html (yes, I have since realized the background is horrible, and changing it is on my list of things to do). Feedback would be appreciated, especially from non-consequentialists and non-philosophers since they're the target audience.

In response to What is Metaethics?
Comment author: Yvain 25 April 2011 09:19:09PM *  12 points [-]

This post covered at least as much material as my old college moral philosophy classes did in a month. It also left me feeling more confident that I understood all the terms involved than that month of classes did. Thank you for being able to explain difficult things clearly and concisely.

In response to comment by Yvain on What is Metaethics?
Comment author: Yvain 26 April 2011 01:30:07AM *  6 points [-]

I request an explanation of why my comment telling Luke he did a good job is more highly upvoted than the post Luke did a good job on. If you agree with me that Luke did a good job strongly enough to upvote the statement, why not upvote Luke?

In response to What is Metaethics?
Comment author: thomblake 25 April 2011 10:00:29PM *  0 points [-]

Tangent: I think Ayer's observation was correct but he had the implication backwards. The English sentence "Yuck!" contains the assertion "That is bad." and is truth-apt.

I have launched into arguments with people after they expressed distaste, and I think it was at least properly grammatical. A start: "What's yucky about that?"

Comment author: Yvain 25 April 2011 10:44:58PM 3 points [-]

When I was in Thailand, I saw some local tribesmen eat a popular snack of giant beetles. I said "Yuck!" and couldn't watch them. However, I recognize that there's nothing weirder about eating a bug than about eating a chicken and that they're perfectly healthy and nutritious to people who haven't been raised to fear eating them.

In response to What is Metaethics?
Comment author: Yvain 25 April 2011 09:19:09PM *  12 points [-]

This post covered at least as much material as my old college moral philosophy classes did in a month. It also left me feeling more confident that I understood all the terms involved than that month of classes did. Thank you for being able to explain difficult things clearly and concisely.

Comment author: Alicorn 25 April 2011 04:46:24PM 1 point [-]

Color vision goes away in dim light, which was readily available before photography.

Comment author: Yvain 25 April 2011 04:52:08PM *  3 points [-]

Er, yeah, but there's no reason people would use dim light as a reference point for their dreams.

When you're trying to remember your dreams, you come up with a bunch of half-recalled images from a short narrative of unusual happenings that has little to do with your everyday life. You parse that as a movie and apply movie conventions to it.

Do you dream in 3D? Your kids will.

Comment author: Psychohistorian 25 April 2011 02:32:12PM *  1 point [-]

Pre-scientific authors tended to assume they dreamed in color, while studies in the first half of the 20th century found very few people who reported dreaming in color.

This is extremely unsurprising, given that the world was in black and white until sometime in the 1930's.

ETA: I'd hope such a patently absurd statement would get people to follow the link. I was evidently wrong. If you follow the link (which is a classic!), you'll see the point was intended to be humorous. My bad.

Comment author: Yvain 25 April 2011 04:45:04PM *  2 points [-]

Pre-scientific authors tended to assume they dreamed in color, while studies in the first half of the 20th century found very few people who reported dreaming in color. In the 1960s, this consensus was overturned, and recent studies show that today, more than 80% of people report that they dream in color. But there are also certain populations that overwhelming report dreaming in black and white.

Well, yes. People thought they dreamed in color until they got a reference class for imagery of not-fully-experienced unusual events - film and TV - which was in black and white. When film and TV became color, dreams became color again too.

Comment author: JohnH 25 April 2011 03:30:18AM *  -2 points [-]

Ok, I was hoping that these would be actual break downs of the beliefs of transhumanists and how such point to the fact that they are a religion. Unfortunately, it was a lot less helpful then that.

Transhumanism, at least as expressed on this site (which hereafter I will just refer to as transhumanism), is as far as I can tell a religion. It is in fact a standard Abrahamic religion in form. Here is my reasoning for saying that:

1,) Belief in the elect. Transhumanists feel that they alone are rational enough to see the truth, to the point that some of them believe that it is better that others are not told the truth.

2.) Belief in God. Transhumanists believe that they will make a Friendly AI that will take the place of God.

3.) Belief in Resurrection. Transhumanists believe in Cryogenics such that when they die they will be frozen and eventually resurrected.

4.) Belief in Immortality. Transhumanists believe in life extension to the point that eventually their bodies will be able to live indefinitely long.

5.) Belief in Eternal Life. Transhumanists believe that eventually uploads will be possible such that even in the case of an unfortunate accident they will be able to survive.

6.) Belief in Theosis/Exaltation. Transhumanists believe that it may become possible to become one with their AI/God

7.) Belief that everyone that disagrees with them is a heretic. Aumann's Agreement theorem has serious problems in it, yet it is used in arguments as though it were true. If someone doesn't hold the “correct” beliefs about AI or other topics it is automatically assumed that they are less capable or less rational than those that do. This in spite of the fact that they may have additional information that is influencing their priors in important ways.

8) Sacred Symbols. At least those that are signed up for Cryonics have symbols that they wear and use to spark interest in their beliefs or debates about their beliefs.

9) Belief in the coming of a messiah - the singularity.

10) Holy Script - The sequences.

Ok, now that I have you fired up, please deconstruct this. Most especially try deconstructing this argument as though you were explaining things to someone that didn't understand the science or philosophic theories that these belief structures are built off of. Realize that many of the people on this site do not have the training to understand these theories as far as I can tell.

Comment author: Yvain 25 April 2011 02:48:20PM *  5 points [-]

Professional golfers also satisfy points 1, 7, 8 and 10.

Belief in the elect: They believe that they are better golfers than others, and that only they are worthy of competing in the PGA Championship; lesser mortals must be relegated to less prestigious competitions and will be kept out of the clubhouse by armed security guards.

Belief that everyone who disagrees with them is a heretic: They laugh at and mock people who think that you should use a 9-iron when your ball is on the green, or that you should start off with a putter on a par-5 hole. These people are automatically assumed to be less skilled at golf than they are.

Sacred Symbols: They frequently carry around bags of golf clubs and wear special clothing; these can spark conversations with non-golfers.

Holy Script: Every two years, the Royal and Ancient Golf Club of St. Andrews publishes a book called "The Rules of Golf"; this is accorded sacred status and anyone who refuses to follow it is expelled from the brotherhood of professional golfers in disgrace.

I maintain that insofar as we tick any of these boxes, we do so for reasons more like the reasons professional golfers tick them than the reasons the Catholic Church ticks them.

2,3,4,5,6,9 take a lot more explanation as they are unusual beliefs among secular groups, but all we can try to do is justify why we believe them. Have you tried reading the Sequences?

Comment author: Alicorn 25 April 2011 01:24:37AM *  12 points [-]

Here's a more straightforward case of being mistaken about one's experience that happened to me:

Schwitzgebel describes somewhere an experiment you can do with a random playing card. Draw it, and hold it facing you at arm's length directly to your left or right while focusing your eyes straight ahead. Slowly move it around in an arc at arm's length, so it goes through your peripheral vision bit by bit, and try to guess if it's red or black.

I tried this, and I had the weirdest experience. I thought I saw it as black(1), and then I realized it was red, and as the card moved, I did not experience a change in its apparent color; it just became plain that that color, the same one, was red.

So either the card looked black and then red without changing its apparent color (implausible), or I was mistaken about my subjective experience somewhere (the initial color perception, or the absence of a change perception).

(1) I don't remember which it was actually, but it was the wrong one.

Comment author: Yvain 25 April 2011 02:23:13PM 5 points [-]

This is more convincing than Luke's examples, but it's still a case of a flaw in between stimulus and perception, rather than one in consciousness.

You think you perceive a quale of red. Then you think you perceive a quale of black. You notice that you never perceived a quale of change. Each time you are correct about which qualia you did or did not perceive.

You would be wrong if you asserted "my conscious perception of color did not change", but you are correct in asserting "I did not consciously perceive the change in my perception of color."

This may make more sense if you think of perception of change as a specific thing which the brain has to detect and register separate from the changing inputs, rather than as a "natural" consequence of stimuli changing.

This might also make more sense if you think of a Photoshop-style gradient, eg a 200 px gradient between red and blue. You can't perceive a change between each individual pixel and the pixel after it, but pixel 1 is definitely red and pixel 200 is definitely blue. You're not wrong about your conscious experiences at any point, your conscious experience just isn't picking up the change very well.

Comment author: Yvain 25 April 2011 12:47:07AM *  14 points [-]

"We can be wrong about our subjective experiences" is a very broad and maybe misleading way of summarizing this information.

You've shown that we can sometimes fail to consciously pick up on certain channels of sensory information (like echolocation) or fail to consciously appreciate certain qualities of our sensory experience (like the ellipticalness of coins or the vagueness of imagery).

But all your examples have the quality that if we consciously think about them, we realize our mistake (I know this is kind of unfair to you as you couldn't have used an example that didn't have that quality because people wouldn't admit they were wrong about it). So this post only re-affirms that there is a conscious-unconscious distinction, and that lots of the things we think are fully conscious processes are the conscious shadows of larger unconscious processes.

When I think of being wrong about subjective experience, I think of being wrong about the "contents" of "my" "own" "consciousness". The problem here isn't that I'm wrong about my consciousness, it's that certain things never entered my consciousness to begin with and were handled by unconscious routines without me noticing. As soon as I think about them consciously, I'm no longer wrong.

This is also how people like Descartes used the idea of subjective experience, and I don't think knowing about these examples would make Descartes hesitate to say with certainty that he can be sure he's really thinking if he thinks he's thinking.

Comment author: Alicorn 24 April 2011 01:54:20AM 3 points [-]

Speaking of the read-the-book-of-Mormon-and-pray-about-it-and-get-a-straight-answer experiment, I've actually told a couple of my friends that I will eventually do this in the name of empiricism, but it's such a profoundly boring book that I haven't gotten very far yet. Is there a way to read it that makes it more interesting? Why isn't scripture better-crafted?

Comment author: Yvain 24 April 2011 01:58:23AM 2 points [-]

Is there a way to read it that makes it more interesting?

Liveblog it. Chapter by chapter.

Comment author: Yvain 21 April 2011 07:04:18PM 7 points [-]

I have the same feeling.

The only caveat I have with skipping pages is that it's a slippery slope. I sometimes feel if I let myself skip the boring parts, I'll gradually start skipping more and more until all I read are the funny anecdotes. This is definitely true of any book I'm reading because I have to (say, for school), less so for books I genuinely enjoy.

In response to Singularity FAQ
Comment author: Yvain 21 April 2011 12:03:49AM *  1 point [-]

This is an excellent singularity FAQ which handles a very difficult task surprisingly well.

Comment author: Yvain 19 April 2011 07:51:09PM 5 points [-]

Why are you reposting a summary+link, instead of the post itself?

Comment author: Psychohistorian 19 April 2011 05:47:45PM *  1 point [-]

I don't think it's ever okay to literally discount utility.

I'm actually (and this is my intro point) not sure it's possible to avoid doing this. If we have some ice cream that you and I both want, we must necessarily engage in some weighing of our interest in the ice cream. There are objective measures we can use (i.e. how much we'd be willing to pay for it; how many hours of labor we'd sacrifice to obtain it, etc.), but I'm fairly confident there is not an Objective Measurement of True Utility that Tells Us Who Absolutely Deserves the Ice Cream. Much utilitarian thinking appears contingent on this philosophical fiction (perhaps this point is itself the primary one). Any selection of an objective criteria either implicitly or explicitly discounts something about one of the agents - willingness to pay favors the rich, willingness to spend time may favor the young or unemployed, etc.

As for Bob the Rapist, the issue is not that he enjoys rape because it hurts other people, but that he is knows it causes harm and doesn't care. This may surprise you, but the vast majority of humanity is not comprised of unweighted aggregate utilitarians. Though I think our actual disagreement may not exist - if he engages in fulfilling rape fantasies with consenting adults, or makes a simulated world for himself, or designs a sex-bot to enjoy being raped (which is itself an ontologically convoluted issue, but I digress), I'm not objecting. So it could be my, "Discounting his utility" is your "dismissal for game-theoretic reasons" are essentially the same thing. If we call my system U' versus standard U, perhaps the argument is that any kind of applied utilitarian framework needs to look more like U' than like U.

Without writing a whole article on the issue, there does appear to be a difference between forcibly raping someone and wearing fur. Off the cuff, I'd guess that this issue is marginal effect. Animal lovers tend to object to the existence of fur coats generally - the step from 1 fur coat or 100 fur coats or 100,000 fur coats is smaller than the step from 0 to one, and they do not "feel" each fur coat, in the same way that a person "feels" being raped.

Comment author: Yvain 19 April 2011 07:10:06PM *  3 points [-]

I'm not disagreeing that crimes are bad, just that this should be stated as saying that whatever utility they gives the perpetrator is overruled by the disutility they give the victim.

This has occasional theoretical implications: for example, if Bob was insane and incapable of realizing that his actions harmed his victim, then we still perform exactly the same calculation and stop him, even though the "he knows it causes harm and he doesn't care" argument is void.

Even if the panda analogy isn't perfect, there are suitably many analogies for acts where two people's utility is in competition for non-evil reasons: for example, if we both want pizza and there's only one slice left, my taking it isn't bad in itself, but if you are hungrier it may be that my (perfectly valid) desire for pizza is a utilitarian loss and should be prevented.

Given that this same principle of "if two people's interests are in conflict, we stop the person with the lower stake from pursuing that interest at the expense of the person with the higher stake" is sufficient to explain why crimes are bad, I don't see why another explanation is needed.

On an unrelated note, I've heard people suggest that it's a bad idea to use rape as an example in a case where any other example is possible because it's an extreme emotional trigger for certain people. I'm going to try to use murder as my go-to example of an immoral hurtful act from now on, on the grounds that it conveniently removes its victims from the set of people with triggers and preferences.

Comment author: Psychohistorian 19 April 2011 05:50:40PM *  3 points [-]

Pure curiosity, in response to the whole non-discounted utility argument.

In a case similar to the beginning to Kill Bill - orderly selling sex with a comatose, brain dead woman - how does your utilitarian calculator come out? Assume (unlike the movie) she is in fact completely gone. Do you simply bite the bullet and agree it's a positive outcome?

Comment author: Yvain 19 April 2011 07:02:59PM *  3 points [-]

I've never seen that movie, and based on your description I'm thinking I probably shouldn't. But I agree with Nick Tarleton - the relevant preferences are those of the people who are not in comas, but know that one day they might be.

Comment author: Yvain 19 April 2011 04:21:25PM *  9 points [-]

I think I disagree with your interpretation on every one of these points!

1: If you're assuming the reason Bob enjoys his crimes is because he's lowering the utility of his victims, you need to make that assumption much more explicit. If Bob only committed his crimes because it makes him feel good, but doesn't feel better knowing that the victim is unhappy because he has committed them, then he's only in the same position as anyone else who wants something that's good for emself but hurts another person. For example, if I want a coat made of panda-bear fur, then this upsets panda-lovers and (if we allow animals to have utility) pandas, but it doesn't allow anyone to disregard my desire - it just means that I might not get it satisfied if the panda-lovers turn out to have more clout.

Even if Bob explicitly enjoys lowering utility, I still find it sketchy to dismiss his desire as an axiom of the system. In a counterfactual world where all desires are completely fixed and non-self-modifiable, Bob's desires should be taken into account (and probably rejected when it turns out they hurt other people more than they help him). In a world more like our own, where desires become more common and intense if there's a possibility of fulfilling them, these desires should be dismissed for game theoretic reasons to discourage any more such desires from ever forming, not just disregarded.

1.5: I can't tell whether you're just acknowledging the existence of punishment, or something different. Yes, we should punish child molesters as a game theoretic action to discourage child molestation in the future. And this punishment has to involve negative utility. But punishment should be specific and limited. If the child molester has already gone to jail, then ey gets the same weight in our calculations as anyone else. And if we do punish em, we do it not because we're using a system that says we can disregard eir utility on general principle, but because a greater good (discouraging future child molestation) overrides it.

2: Here I would say the villagers don't have a preference against the girl living, they have a preference against witchcraft. We take all of their preferences into account, but because drowning the girl doesn't reduce witchcraft, their preferences have no bearing on whether we save the girl or not. So since there's massive disutility (death) and minimal utility (they get small utility from not having to waste time worrying about witchcraft, but they don't get what they were hoping for which is less witchcraft occuring), it's a net loss and we save the girl. In the ritual dunking case, there's no disutility, and the utility is that they stop worrying about witchcraft, so it's a net gain and we let them do it.

(this makes more sense if we assign numbers: say having less witchcraft is +50, killing a girl is -25, not having to worry about witchcraft so much is +5, and ritually dunking is -1. These assignments make more sense if you imagine asking the villagers "If this girl wasn't a witch, would you want her to continue living?" - their answer is the appropriate preference)

I don't think it's ever okay to literally discount utility. I think it's often okay to let one source of utility counterbalance and override another source of utility, and that sometimes that first source may be a game theoretic precommitment to ignore someone's utility, but you don't stick ignoring someone's utility into the theory directly.

Comment author: knb 16 April 2011 02:43:07AM 14 points [-]

I think the world is better off without sacred cows, rather than with them. The only way to eliminate these kinds of reactions is via "exposure therapy". Admittedly, I say this as someone without many sacred cows. I'm non-religious, an anti-nationalist, and (other than a long career as a "non-denominational" anti-war activist) essentially apolitical.

I support the Mohammed drawing day, Koran-burning, and similar attempts involving other religions and political doctrines. When people do these things, it helps create a safe space for people to speak their reasoned criticisms.

Comment author: Yvain 17 April 2011 12:41:40PM 8 points [-]

I think the world is better off without sacred cows, rather than with them. The only way to eliminate these kinds of reactions is via "exposure therapy"...I support the Mohammed drawing day, Koran-burning, and similar attempts involving other religions and political doctrines.

What about calling black people the n-word, making Holocaust jokes to Jews, and insulting people's dead relatives?

I mean, all these things feel like they're in a different category than the things you described, but I wouldn't know how to describe that difference to a computer.

Comment author: BillyOblivion 17 April 2011 12:02:11PM 1 point [-]

Not true.

I've never been Jewish, and I personally find the act of deliberately desecrating objects of belief, faith and following--from blowup up Buddhas to burning flags to pissing on crucifixes to be extremely distasteful in ways ranging from mild annoyance at juvenile attempts to shock (Piss christ, Dung Madonna) to rather enraging (Blowing up thousand year old statues or burning flags as a political protest).

Things like the Torah, the Koran and the Bible (a bit less so, but still) (I have no idea about the Shruti/Smirtis, but I'll lump them in here anyway) are not just religious texts, they are cultural icons, relics and touchstones to hundreds of millions of people. By desecrating those artifacts you are desecrating those people's beliefs and culture. This is, at minimum rude and is a mind killer.

Now things like shitting on a torah, blowing up a church or burning a flag are distinctly different than drawing Mohammad in that drawing Mohammad (or anyone else really) is not desecration, but blasphemy. It is roughly the same as me, or anyone else "taking the lords name in vain".

Comment author: Yvain 17 April 2011 12:35:23PM 4 points [-]

Now things like shitting on a torah, blowing up a church or burning a flag are distinctly different than drawing Mohammad in that drawing Mohammad (or anyone else really) is not desecration, but blasphemy. It is roughly the same as me, or anyone else "taking the lords name in vain".

Can you explain what criteria you're using to draw that distinction? Do you expect people with different cultural norms to be able to acknowledge those criteria as objectively valid?

Comment author: drethelin 16 April 2011 05:28:47AM 14 points [-]

The slippery slope that applies is not that every random religion will taboo a certain activity, but that the more power you give to one religion the easier it is for it to get even more power. If, through outrageous overreaction, they can force people to stop one activity, they have zero incentive to not use this tactic against everything they are morally against, much of which is of MUCH greater utility than images of Mohammed.

Comment author: Yvain 17 April 2011 12:30:51PM 3 points [-]

I acknowledge that slippery slope argument has some validity, but what I haven't seen so far is a good criterion of where to apply it.

I feel confident being upset at people who draw swastikas where they expect Jews to see them, or burn crosses outside the houses of black people. Although one can make all the same arguments ("if we embolden the blacks and Jews by giving in now, then they'll start demanding more and more rights until we have to believe as they do") I still think doing either of those actions is wrong.

Drawing Mohammed seems designed to harass Muslims in the same way that drawing swastikas seems designed to harass Jews. So where is the critical difference that makes one necessary and the other abhorrent?

Comment author: Alicorn 16 April 2011 09:40:00PM 6 points [-]

How should this interact with people who are interested in seeing the display in question? (E.g. I once made out with a girl on a bus full of people and we got lots of, er, positive attention. How should I have weighted that vs. your discomfort with public displays of gay affection if you had been on the bus with us?)

Comment author: Yvain 16 April 2011 09:44:29PM *  4 points [-]

I just realized that when I said "gay", I meant "gay male".

Although to answer your question, you'd have to sum up the positive and negative preferences of people who might see you. I expect you'd probably be in the clear at a college pub, less so at the Retired Baptist Womens' Convention.

Comment author: Vladimir_M 16 April 2011 09:29:25PM *  4 points [-]

I still believe that if you find something offensive, a request to change phrased in the language of harm-minimization is better than a demand to change phrased in the language of offense, but I don't know if anyone is challenging that.

I see at least two huge problems with the harm-minimization approach.

First, it requires interpersonal comparison of harm, which can make sense in very drastic cases (e.g. one person getting killed versus another getting slightly inconvenienced), but it usually makes no sense in controversial disputes such as these.

Second, even if we can agree on the way to compare harm interpersonally, the game-theoretic concerns discussed in this thread clearly show that naive case-by-case harm minimization is unsound, since any case-by-case consequences of decisions can be overshadowed by the implications of the wider incentives and signals they provide. This can lead to incredibly complicated and non-obvious issues, where the law of unintended consequences lurks behind every corner. I have yet to see any consequentialists even begin to grapple with this problem convincingly, on this issue or any other.

Comment author: Yvain 16 April 2011 09:32:24PM 2 points [-]

We may be talking at cross-purposes. Are you arguing that if someone says something I find offensive, it is more productive for me to respond in the form of "You are a bad person for saying that and I demand an apology?" than "I'm sorry, but I was really hurt by your statement and I request you not make it again"?

Comment author: Desrtopa 16 April 2011 02:40:20PM *  14 points [-]

Muslims' sensitivity to Mohammed is based on a falsehood; Islam is a false religion and Mohammed is too dead to care how anyone depicts him. I agree with this statement, but I don't think it licenses me to cause psychic pain to Muslims. I couldn't go around to mosques and punch Muslims in the face, shouting "Your religion is false, so you deserve it!".

This strikes me as a bad analogy. Seeing pictures of Mohammed is only offensive to Muslims because of their conviction in a poorly evidenced falsehood, whereas punching someone in the face is an offense regardless of what they believe. I think that a more apt comparison would be holding communion wafers hostage in order to offend Catholics.

If I thought that actions like these would discourage people from taking offense due to falsehoods, I would consider that to be a strong argument in their favor, but I don't see that they're actually doing much aside from fueling persecution complexes and feeding conflict.

Comment author: Yvain 16 April 2011 09:27:40PM *  7 points [-]

How's this for a metaphor: suppose I thought my mother had died in the Holocaust, when in fact she'd escaped the Holocaust without incident and simply lost contact with me. Someone makes Nazi jokes around me, or says that everyone who died in the Holocaust deserved it and went to Hell, or something equally offensive.

Suppose my interlocutor knows that my mother did not die in the Holocaust, and knows that if I believed my mother didn't die in the Holocaust I wouldn't be offended by what ey's saying. Ey also knows that since I do believe my mother died in the Holocaust, I definitely will be offended.

Even in this situation - in which I am only suffering because I have a false belief, and for reasons directly related to that false belief - I still think my interlocutor is very much in the wrong.

Comment author: Psychohistorian 16 April 2011 05:18:22AM *  20 points [-]

A couple points.

You miss an important issue, which is the western concept as speech as a right. The Folsom street fair can have a promo poster of the last supper as Jesus as a naked black dude surrounded by transvestites, dominatrixes, and sex toys, and no major Christian organization will propose that anyone should be killed. They may try to get funding taken away from the fair, but that's their right. Westerners have a concept of appropriate levels of conflict, and if someone violates them, we want to punish them. If someone asks me politely to keep it down, I probably will. If they tell mento shut up or they'll kick my ass, my instinct is to talk even louder (especially if they're bluffing). This is sensible as annoy of punishing improper behavior.

I also take issue with your characterization of offense as pain. In some cases - where it's directed at someone, like racial slurs, it is. But in cases of taking offense at untethered actions, pain isn't accurate. It's not exactly painful when, say, a Klansmen sees an interracial couple, even if he finds their behaviour offensive. And even if it were, it seems obvious to menthat the couple should not allow that to affect their behaviour. If the Brits in your example just got arbitrarily angry about seeing trout pictures, I'm not sure the same reaction follows. Perhaps if you taboo offense, you get a more coherent picture of two separate emotional reactions.

Comment author: Yvain 16 April 2011 09:21:18PM *  7 points [-]

I would like to believe the Klansman (I was considering changing this to Klansperson, but political correctness is probably inappropriate in this situation) doesn't feel anything like real suffering when he sees an interracial couple, but I have no evidence for this except my desire to sweep his feelings under the rug so I don't have to use them in ethical calculus.

For example, I am strongly pro gay rights and gay marriage, but I admit that seeing public displays of affection between gays gives me a negative visceral reaction more than the same displays among straights do. If I could self-modify to remove this feeling I'd do so in a second, but given that I can't self-modify it seems like this preference is worthy of utilitarian respect; eg insofar as they want to be nice to me, gay people should avoid PDAs around me when it's not too inconvenient for them (and if gay people have the same feeling in reverse, straight people who are nice should avoid hetero PDAs around them).

I have no reason to think I can model Klansmen well, but when I try, I imagine their feelings around an interracial couple as being a lot like my feeling around gay people having PDAs.

Comment author: HughRistik 16 April 2011 08:29:26PM 14 points [-]

Voluntary self-modification also requires a pre-existing desire to self-modify.

People have motives to increase their status, so we can check this box. Of course, this depends on phenotype, and some people do this much more than others.

I wouldn't take a pill that made me want to initiate suicide attacks on people who insulted the prophet Mohammed, because I don't really care if people insult the prophet Mohammed enough to want to die in a suicide attack defending him.

You can't self-modify to an arbitrary belief, but you can self-modify towards other beliefs that are close to yours in belief space. See my comment about political writers. You can seek out political leaders, political groups, or even just friends, with beliefs slightly more radical than yours along a certain dimension (and you might be inspired to do so with just small exposure to them). Over time, your beliefs may shift.

If X doesn't offend you, why would self-modify to make X offend you to stop people from doing X, since X doesn't offend you?

To protect/raise the status of you yourself, or of a group you identify with. I proposed in that comment that people might enjoy feeling righteous while watching out for the interests of themselves and their in-group. When you get mad about stuff and complain about it, you feel like you are accomplishing something.

Thus, someone who responded with a cost/benefit calculation to all respectful and reasonable demands to stop offending, but continued getting touchy about disrespectful blame-based demands to stop offending, would be pretty hard to game.

The problem is that other people only care if you are with them or against them; they don't care about your calculation.

The second problem is that it can be hard to distinguish these two things. People who have a sufficiently valid beef might be justified in making blame-based demands to stop offending, and your demand that they sound "respectful" and "reasonable" is itself unreasonable. Of course, people without a valid beef will use this exact same reasoning about why you can't make a "tone argument" against them asking for them to sound more respectful and reasonable.

There might be a correlation between offense and the "validity" of the underlying issue, but this correlation is low enough that it can be hard to predict the validity of the underlying issue from how the offense reaction is expressed, which weakens the utility of the strategy you propose for identifying beefs.

However, your strategy might be useful as a Schelling Point for what sort of demands you'll accept from others.

One difference between this post and the original essay I wrote which more people liked was that the original made it clearer that this was more advice for how people who were offended should communicate their displeasure, and less advice for whether people accused of offense should stop.

It may have been tough to get the message, because the British salmon example is hypothetical. A real-world example of some group succeeding in claims of offensive might be useful.

Comment author: Yvain 16 April 2011 08:43:34PM *  40 points [-]

Okay. I formally admit I'm wrong about the "should usually stop offensive behavior" thing (or, rather, I don't know if I'm wrong but I formally admit my previous arguments for thinking I was right no longer move me and I now recognize I am confused.)

I still believe that if you find something offensive, a request to change phrased in the language of harm-minimization is better than a demand to change phrased in the language of offense, but I don't know if anyone is challenging that.

Comment author: RichardKennaway 16 April 2011 06:59:02AM 14 points [-]

My first reaction to this was "is this a guest post by Robin Hanson under Yvain's name, to see if anyone notices?"

You could argue Brits did not choose to have their abnormal sensitivity to salmon while Muslims might be considered to be choosing their sensitivity to Mohammed. But this requires a libertarian free will.

Well, no it doesn't. Muslims observably do make a choice in the matter (as proved by the fact that they discuss it and take different views). (Link.) To equate this with aliens hard-wiring our brains to graft on an arbitrary offense-trigger is plain no-free-will determinism, whereby the past reaches past the present to cause the future, just as the alien reaches past our internal functions to cause offense-taking at an arbitrary stimulus.

Comment author: Yvain 16 April 2011 07:40:53PM 6 points [-]

My first reaction to this was "is this a guest post by Robin Hanson under Yvain's name, to see if anyone notices?"

Part of me wants to feel complimented by that, another part wants to challenge you to pistols at dawn.

Comment author: Vladimir_M 16 April 2011 02:04:38AM 28 points [-]

Another unpleasant implication of the consequentialist attitude towards offense is that societies should be as homogeneous as possible with regards to people's values and beliefs. (And I'm not talking about Aumann-agreement here!) As the diversity of a society increases, the set of statements and acts that can be done in public without offending one group or another necessarily shrinks, which implies an inevitable trade-off between the pain of offense and the pain of people who have their freedom curtailed and are increasingly forced to walk on eggshells. I'll leave the more concrete implications in the context of today's politics as an exercise for the reader.

It also implies that a certain level of isolation between societies is desirable, in direct opposition to the present trends of globalization. What is regular business in one society may well be extremely offensive in another. So, if there's an intense mutual interest and exchange of information between societies, we get the same problem as within a single diverse society. This can be mitigated only by isolating these societies from each other so that their members are not exposed to the painful sight of the offensive alien customs.

Comment author: Yvain 16 April 2011 07:39:12PM *  9 points [-]

All of this seems pretty true to me. There were even studies that showed pretty clearly that ethnically homogenous communities were happier than ethnically mixed ones.

There are lots of good reasons not to actually exclude different people from a society. Immigration's been shown to be a net good for most people involved, and of course uprooting people from a society they've grown accustomed to is harmful. But these only counterbalance the above claim, not disprove it.

I think it's pretty self-evident that anything that brings nudists together with those Arabs who freak out if every inch of a woman isn't covered by a burka is going to be a net loss for both groups.

Comment author: Vladimir_M 16 April 2011 01:50:28AM *  98 points [-]

Yvain:

The offender, for eir part, should stop offending as soon as ey realizes that the amount of pain eir actions cause is greater than the amount of annoyance it would take to avoid the offending action, even if ey can't understand why it would cause any pain at all.

In a world where people make decisions according to this principle, one has the incentive to self-modify into a utility monster who feels enormous suffering at any actions of other people one dislikes for whatever reason. And indeed, we can see this happening to some extent: when people take unreasonable offense and create drama to gain concessions, their feelings are usually quite sincere.

You say, "pretending to be offended for personal gain is... less common in reality than it is in people's imaginations." That is indeed true, but only because people have the ability to whip themselves into a very sincere feeling of offense given the incentive to do so. Although sincere, these feelings will usually subside if they realize that nothing's to be gained.

Comment author: Yvain 16 April 2011 07:30:15PM *  2 points [-]

I'm not sure people can voluntarily self-modify in this way. Even if it's possible, I don't think most real people getting offended by real issues are primarily doing this.

Voluntary self-modification also requires a pre-existing desire to self-modify. I wouldn't take a pill that made me want to initiate suicide attacks on people who insulted the prophet Mohammed, because I don't really care if people insult the prophet Mohammed enough to want to die in a suicide attack defending him. The only point at which I would take such a pill is if I already cared enough about the honor of Mohammed that I was willing to die for him. Since people have risked their lives and earned lots of prison time protesting the Mohammed cartoons, even before they started any self-modification they must have had strong feelings about the issue.

If X doesn't offend you, why would self-modify to make X offend you to stop people from doing X, since X doesn't offend you? I think you might be thinking of attempts to create in-group cohesion and signal loyalty by uniting against a common "offensive" enemy, something that I agree is common. But these attempts cannot be phrased in the consequentialist manner I suggested earlier and still work - they depend on a "we are all good, the other guy is all evil" mentality.

Thus, someone who responded with a cost/benefit calculation to all respectful and reasonable demands to stop offending, but continued getting touchy about disrespectful blame-based demands to stop offending, would be pretty hard to game.

One difference between this post and the original essay I wrote which more people liked was that the original made it clearer that this was more advice for how people who were offended should communicate their displeasure, and less advice for whether people accused of offense should stop. Even if you don't like the latter part, I think the advice for the former might still be useful.

Offense versus harm minimization

60 Post author: Yvain 16 April 2011 01:06AM

Imagine that one night, an alien prankster secretly implants electrodes into the brains of an entire country - let's say Britain. The next day, everyone in Britain discovers that pictures of salmon suddenly give them jolts of painful psychic distress. Every time they see a picture of a salmon, or they hear about someone photographing a salmon, or they even contemplate taking such a picture themselves, they get a feeling of wrongness that ruins their entire day.

I think most decent people would be willing to go to some trouble to avoid taking pictures of salmon if British people politely asked this favor of them. If someone deliberately took lots of salmon photos and waved them in the Brits' faces, I think it would be fair to say ey isn't a nice person. And if the British government banned salmon photography, and refused to allow salmon pictures into the country, well, maybe not everyone would agree but I think most people would at least be able to understand and sympathize with the reasons for such a law.

So why don't most people extend the same sympathy they would give Brits who don't like pictures of salmon, to Muslims who don't like pictures of Mohammed?


SHOULD EVERYBODY DRAW MOHAMMED?

I first1 started thinking along these lines when I heard about Everybody Draw Mohammed Day, and revisited the issue recently after discovering http://www.reddit.com/r/mohammadpics/.

I have to admit, I find these funny. I want to like them. But my attempts to think of reasons why this is totally different from showing pictures of salmon to British people fail:

• You could argue Brits did not choose to have their abnormal sensitivity to salmon while Muslims might be considered to be choosing their sensitivity to Mohammed. But this requires a libertarian free will. Further, I see little difference between how a Muslim "chooses" to get upset at disrespect to Mohammed, and how a Westerner might "choose" to get upset if you called eir mother a whore. Even though the anger isn't being caused by alien technology, it doesn't feel like a "choice" and it's more than just a passing whim. And if tomorrow I tried to "choose" to become angry every time someone showed me a picture of a salmon, I couldn't do it - I could pretend to be angry, but I couldn't make myself feel genuine rage.

• Muslims' sensitivity to Mohammed is based on a falsehood; Islam is a false religion and Mohammed is too dead to care how anyone depicts him. I agree with this statement, but I don't think it licenses me to cause psychic pain to Muslims. I couldn't go around to mosques and punch Muslims in the face, shouting "Your religion is false, so you deserve it!".

• It is necessary to draw pictures of Mohammed to show Muslims that violence and terrorism are inappropriate responses. I think the logic here is that a few people drew pictures of Mohammed, some radicals sent out death threats and burned embassies, and now we need to draw more pictures of Mohammed to convince Muslims not to do this. But it sounds pretty stupid when you put it in exactly those words. Say a random Christian kicked a Muslim in the face, and a few other Muslims got really angry, blew the whole thing out of proportion, and killed him and his entire family. This would be an inappropriately strong response, and certainly you could be upset about it, but the proper response wouldn't be to go kicking random Muslims in the face. They didn't do it, and they probably don't even approve. But drawing pictures of Mohammed offends many Muslims, not just the ones who send death threats.

• The slippery slope argument: if we allow Muslims' concerns to prevent us from drawing pictures of Mohammed, sooner or later we'll have to accept every two-bit group with a ridiculous superstition and we'll never be able to get anything done. I take this more seriously than the previous three arguments, but I've previously argued that granting large established religions special rights is relatively immune to slippery-slope. And anyway, drawing pictures of Mohammed is such an unusual thing to do that we can stop doing it without giving up our right to keep doing something else that's actually useful if the situation comes up later.

None of these excuses really does it for me. So my provisional conclusion is that yes, people who draw pictures of Mohammed where Muslims can see them are bad people in the same way that people who go around showing photos of salmon to Brits are bad people.

So the big question is: why is this so controversial in the Mohammed example, when it seems so obvious in the salmon example?

A BLAME-BASED CONCEPT OF OFFENSE

I think several features of the salmon example trigger consequentialist moral reasoning, in which the goal is to figure out how to satisfy as many people's preferences as possible; several contrasting features of the Mohammed case trigger deontological moral reasoning, in which the goal is to figure out who is a good person or a bad person and to assign status and blame appropriately. These two forms of reasoning give different results in the two different cases.

The word that comes up a lot in discussions of this sort of issue is "offensive". When someone draws Mohammed, it is considered offensive to Muslims. When someone writes a story where all the sympathetic and interesting characters are male, it is considered offensive to women.

For me, the word "offensive" brings up connotations of "It was morally wrong to say this, and you are either inexcusably ignorant of this fact or deliberately malicious. You must immediately apologize, and it is up to the group you have offended to decide whether they accept your apology or whether they want to punish you in some well-deserved way."

This means that ever admitting you were offensive is a huge status hit implying you are some combination of callous, ignorant, and racist. Sometimes people may be willing to take this status hit, especially if upon reflection they believe they really were in the wrong, but since most people's actions seem reasonable to themselves they will not be willing to accept a narrative where they're the villain.

More likely, they will try to advance an alternative interpretation, in which their actions were not legitimately offensive or in which they have the "right" to take such actions. Such an interpretation may cast the offended party as a villain, trying to gain power and control by pretending to be offended, or unduly restricting the free speech of others.

The controversy over drawing Mohammed has several factors that predispose to this sort of interpretation. There is already a history of misunderstanding and some enmity between Muslims and non-Muslims. Muslims' status as a minority makes ideas of "political correctness" readily primed and available, making people likely to miss the trees for the forest. Muslims are often of a different race than Christians, so conflicts with them risk tarring a person with the deeply insulting label of "racist". And because there are reports of Muslims rioting and hurting other people because of Mohammed drawings, they are easy to villainize.

This risks embroiling everyone in an unproductive argument about whether an action was "legitimately offensive" or not, with much status riding on the result.

A CONSEQUENTIALIST CONCEPT OF HARM MINIMIZATION

The British salmon example, on the other hand, was designed to avoid the idea of "offense" and trigger consequentialist notions of harm minimization2.

The example specifically refers to the displeasure that salmon cause the British as "psychic pain", priming ideas about whether it is acceptable to cause pain to another person. The British are described as politely asking us to avoid salmon photography as a favor to them, putting themselves in a low status position rather than demanding we respect their status. British are white and first world, so it's hard to think of this as a political correctness issue and wade into that particular quagmire. And because the whole salmon problem is the result of an alien prankster, there's no easily available narrative in which the British are at fault.

A consequentialist reasoner would consider how much disutility it causes not to be able to use pictures of salmon where the British might see them, then consider how much disutility it causes the British to see pictures of salmon, and if the latter outweighed the former, they'd stop with the salmon pictures. There's an argument to be made about slippery slope, but in this case the slope doesn't seem too slippery and other cases can be evaluated on their merits.

And a consequentialist British person, when considering how to convince a foreigner to stop using pictures of salmon, would try to phrase eir request in a way that minimizes the chances that the foreigner gets upset and confrontational, and maximizes the chances that they actually stop with the salmon.

If the foreigner refused to stop with the salmon pictures, the British person would try to shame and discredit the foreigner into doing so only if ey thought it would work better than any less confrontational method, and only if the chance of it successfully stopping the offending behavior was great enough that it outweighted the amount of bad feelings and confrontation it would cause.

This is a healthier and potentially more successful method of resolving offensive actions.

OFFENSE AND TYPICAL MIND FALLACY

I post on a forum where a bunch of regulars recently denounced the culture of verbal abuse. The abusers, for their part, said that the victims were making mountains out of molehills: exaggerating some good-natured teasing in order to look holier-than-thou.

I was friends with some of victims and with some abusers; neither side were majority bad people, and it surprised me that people would view requests to stop verbal abuse as a Machiavellian ploy.

Not to say that asking for verbal abuse to stop can't be a Machiavellian ploy. In fact, as far as Machiavellian ploys go, it's a pretty good one - take something your political enemies do, pretend to be deeply offended by it, and then act upset until your enemies are forced to stop, inconveniencing them and gaining you sympathy. A conspiracy such is this is not impossible, but why is it so often the first possibility people jump to?

I think it has to do with something I heard one of the abusers say: "I would never get upset over something little like that."

I know him and he is telling the truth. When someone is verbally confrontational with him, he takes it in stride or laughs it off, because that's the kind of guy he is.

I am of Jewish background. I've had someone use an anti-Semitic slur on me exactly once. My reaction was the same mix of confusion and amusement I'd feel if someone tried a vintage Shakespearean insult. And yet I also know of Jews who have been devastated by anti-Semitic slurs, to the point where they've stopped going to school because someone in school taunted them. These people may differ from me in terms of Jewish identity, extraversion, demographics, social status, anxiety, neurogenetics, and some hard-to-define factor we might as well just call "thin skin".

The point is, if I use my own reactions to model theirs, I will fail, miserably. I will try to connect their reaction to the most plausible situation in which my mind would generate the same reaction in the same situation - in which I am not really upset but am pretending to be so for Machiavellian motives.

In the case of anti-Semitism, it's easy to see factors - like a history of suffering from past prejudice - that make other people's responses differ from mine. It's less obvious why someone else might differ in their response to being called ugly, or stupid, or just being told to fuck off - but if these differences really exist, they might explain why people just can't agree about offensive actions.

A thick-skinned person just can't model a person with thinner skin all that well. And so when the latter gets upset over some insult, the thick-skinned person calls them "unreasonable", or assumes that they're making it up in order to gain sympathy. My friends in the online forum couldn't believe anyone could really be so sensitive as to find their comments abusive, and so they ended up doing some serious mental damage.

SUMMARY

Consequentialism suggests a specific course of action for both victims of offense and people performing potentially offensive actions. The victim should judge whether ey believes the offense causes more pain to em than it does benefit to the offender; if so, ey should nonjudgmentally request the offender stop while applying the Principle of Charity to the offender, and if ey wants the maximum chance of the offense stopping, ey should resist the urge to demand an apology or do anything else that could potentially turn it into a status game.

The offender, for eir part, should stop offending as soon as ey realizes that the amount of pain eir actions cause is greater than the amount of annoyance it would take to avoid the offending action, even if ey can't understand why it would cause any pain at all. If ey wishes, ey may choose to apologize even though no apology was demanded.

If the offender refuses, the victim should only then consider "punishment" by trying to shame the offender and make em appear low status, and only if ey thinks this has a real chance of stopping the offending behavior either in this case or in the future. Like all attempts to deliberately harm another person, this course of action requires of the victim exceptional certainty that ey is in the right.

Although people pretending to be offended for personal gain is a real problem, it is less common in reality than it is in people's imaginations. If a person appears to suffer from an action of yours which you find completely innocuous, you should consider the possibility that eir mind is different from yours before rejecting eir suffering as feigned.

 

FOOTNOTES

1) Thanks to Kaj Sotala, Vladimir Nesov, and kovacsa-whose-LW-name-I-don't-know for originally encouraging me to turn the original essay into an LW post.

2) The deontological notion of offense doesn't really supervene on an idea of pain to other people. If two white people, talking where no black people could possibly overhear them, make a racist joke about black people, that is still "offensive", because racism is wrong no matter what. A consequentialist notion of offense could better ground such examples by theorizing that whites telling racist jokes to other whites creates a climate in which racism is considered acceptable, which eventually will end up hurting someone directly. Or it could decide not to, if it decided the link was too tenuous and hokey - but now any disagreement on the matter is honest disagreement about empirical facts and not philosophical disagreement about who's a bad person.

Comment author: Yvain 13 April 2011 11:20:26PM 2 points [-]

On a related note, people make better decisions (for some values of "better") when their bladders are full. Go figure.

Comment author: dfranke 12 April 2011 03:53:00AM 1 point [-]

I think I can justly paraphrase you as follows:

The gravity, and the qualia, are occurring inside the simulation. You only need to worry about having an actual sphere, or an actual brain, if you want to have effects outside the simulation.

If this paraphrasing is accurate, then I ask you, what does "occurring inside the simulation mean"? What is the physical locus at which the gravity and qualia are happening? I see two reasonable answers to this question: either, "at the simulator", or "nowhere". In the former case, I refer you back to my previous reply. In the latter case, you concede that neither the gravity nor the qualia are real.

Comment author: Yvain 13 April 2011 10:34:34AM *  13 points [-]

Your position within our universe is giving you a bias toward one side of a mostly symmetrical situation.

Let's throw out the terms "real" and "simulated" universe and call them the "parent" and "child" universe.

Gravity in the child universe doesn't affect the parent universe, true; creating a simulation of a black hole doesn't suck the simulating computer into the event horizon. But gravity in the parent universe doesn't affect the child universe either - if I turn my computer upside-down while playing SimCity, it doesn't make my Sims scream and start falling into the sky as their city collapses around them. So instead of saying "simulated gravity isn't real because it can't affect the real universe", we say "both the parent and child universes have gravity that only acts within their own universe, rather than affecting the other."

Likewise, when you say that you can't point to the location of a gravitional force within the simulation so it must be "nowhere" - balderdash. The gravitational force that's holding Sim #13335 to the ground in my SimCity game is happening on Oak Street, right between the park and the corporate tower. When discussing a child-universe gravitational force, it is only necessary to show it has a location within the child-universe. For you to say it "doesn't exist" because you can't localize it in your universe is as parochial as for one of my Sims to say you don't exist because he's combed the entire city from north to south and he hasn't found any specific location with a person named "dfranke".

Comment author: Louie 11 April 2011 04:32:22AM 22 points [-]

newbies to the site in particular, they're not going to see all the awesomeness of this site if they simply look at the frontpage and think it's a community just for meetups.

Even without meetups, the promoted feed is still a pretty terrible way to browse the site for a newbie. It would be like if Wikipedia's main page was the recently created pages. LW needs a homepage. I've asked for a volunteer to help create one and promised to implement it if it's designed. I'm not a designer but I'm offering to help with the implementation... but no one has even tried.

There's karma for complaining, but none for fixing problems. I think when Yvain, the LW admins, and I helped create the entire discussion sub-reddit, we got 3 karma apiece in comments related to it. You've already gotten more karma just for complaining about meet-up announcements for the 10th time without offering to help.

Comment author: Yvain 11 April 2011 12:10:37PM *  24 points [-]

Have you considered a top-level post, possibly by you but even better by Matt from Tricycle, saying:

"Hello! I am the official contact for Less Wrong administration. If you have something you want to code for the site, message me and I will tell you whether you have official permission and whether it will get added when you are done. The answer is probably yes.

Here is the language you should code in, the standards you should follow, and so on. Here's where you can find the rest of the open-source code to see what you should be integrating it with. And here is the process to get your code running and up on the site when it is done, even if that process is just "email it to me".

In the comments to this post, please discuss your programming projects for Less Wrong, check that they're community-supported, and recruit people to help with them. If you can't program, this would also be a good place to suggest projects and recruit people who can."

Then stick a permalink to it similar to the "About" link going to Eliezer's first explanatory post.

All this information is sort of available right now, but not in one place and not with explicit permission to work on things, so the trivial inconveniences add up.

Comment author: Yvain 10 April 2011 01:49:53PM *  10 points [-]

I am very inexperienced in epistemology, so forgive me if I'm making a simple error.

But it sounds like everything important in your theory is stuck into a black box in the words "criticize the idea".

Suppose we had a computer program designed to print the words "I like this idea" to any idea represented as a string with exactly 5 instances of the letter 'e' in it, and the words "I dislike this idea because it has the wrong number of 'e's in it" to any other idea.

And suppose we had a second computer program designed to print "I like this idea" to any idea printed on blue paper, and "I dislike this idea because it is on the wrong color paper" to any idea printed on any other color of paper.

These two computers could run through your decision making process of generating and criticizing ideas, and eventually would settle on the first idea generated which was written on blue paper and which used the letter 'e' exactly five times.

So it would seem that for this process to capture what we mean by "truth", you have to start out with some reasoners who already have a pretty good set of internal reasoning processes kind of like our own that they use when criticizing an idea.

But everything that's interesting and difficult about epistemology is captured in that idea of "a pretty good set of internal reasoning processes kind of like our own that they use when criticizing an idea", so really this decision-making process only works for entities that are already running a different epistemology that's doing all the work.

It almost seems like a detached lever fallacy, where the lever is the ability to criticize ideas, and the machinery the lever is activating is the actual epistemology the agent is using.

Comment author: Yvain 09 April 2011 04:03:47PM *  19 points [-]

I got to shadow some breast physicians last month, and although it's sort of off topic I think I gained some insight as to why so many doctors get this question wrong.

Which is because it's very different from any situation they ever come across in clinical practice. Guidelines are to screen people with mammography and examination; anyone who comes up as suspicious on those two tests then gets a biopsy. No one gets diagnosed with breast cancer from a mammogram alone, the progression from mammogram on to the next step is hard-coded into a pre-determined algorithm, and so the question of "This woman got a positive on the mammogram; does she have cancer?" never comes up. A question that does come up a lot is a woman panicking because she got a positive mammogram and demanding to know whether she has breast cancer, and the inevitable answer is "We'll need to do more tests, but don't worry too much yet because most of these things are false positives."

So the doctors involved know that most real mammogram results are false positives, they know how to diagnose breast cancer based on the combination of tests they actually do, they just can't do Bayesian math problems when given probabilities. This is kind of interesting if you're curious about their intelligence but as far as I know doesn't really affect clinical care.

Comment author: jschulter 09 April 2011 12:14:32AM 0 points [-]

Were you planning on running the game in person, or would there be a chance of doing it remotely. I've only had a little experiences with role-playing games, but I enjoyed it quite a lot.

Comment author: Yvain 09 April 2011 03:47:08PM 1 point [-]

Remotely, but I haven't figured out the details yet.

Comment author: jwhendy 04 April 2011 03:28:29PM 3 points [-]

I have had some success in writing very long and complete essays...and putting it on a blog or website.

Good to know. I have been entertaining that idea as well and started trying to make it real at my blog.

Then when someone asks why I believe X, I just tell them I'll send them a link to the essay.

This is extremely appealing. While further debate might arise later, I think this would quite defuse the situation and avoid the pitfalls of on-the-spot debates (especially since person-to-person discussion almost always lacks the ability to provide sources).

I have had good experiences with religious people by confounding as many atheist stereotypes as possible...

Interesting tactic! I'll have to ponder this one. In my circles, the Lewis trilemma is still thought to hold and they don't think very fondly of the Jesus-as-great-teacher crew.

...not bringing the issue up but having a few overwhelmingly strong points that they will agree with when it is brought up...

Could you expand on this? I'm not sure I understand what overwhelmingly strong points you might bring up that your opponent might agree with. Would this be something like priest scandals? Or not having your prayers for understanding answered?

I also find that if my goal is just to end an argument without losing too much social capital or coming across as confrontational, I get better results with emotional rather than intellectual points...

Another interesting strategy I'll think further about. I'm tempted to think I've already adopted this sort of strategy, though more so through being overly "hazy." Earlier, I would go into far more details, whereas now I've found that if I just say that "I'm not convinced," and offer as few supporting details as possible, I do end up at your example destination: person shrugs, presents puppy-dog stare of pity/compassion (fine line...), and says they'll pray for me.

Thanks for the response; there are some great points to ponder here and perhaps this is the encouragement I need to finally write my "summary of non-belief."

Comment author: Yvain 09 April 2011 03:46:38PM 2 points [-]

Could you expand on this? I'm not sure I understand what overwhelmingly strong points you might bring up that your opponent might agree with. Would this be something like priest scandals? Or not having your prayers for understanding answered?

The one I mentioned about the Holocaust would be my go-to example. But really what's important is that it's not something completely intellectual they're going to have a cached response for.

On a side note, I've never understood people who use priest scandals as evidence for atheism. It seems totally ad hominem - "some of the guys who talk about this God stuff are bad people, therefore it's wrong". I guess you could get there by saying that if God existed He wouldn't allow such evil among His followers, but the only possible response to that would be "And where have you been for the past five thousand years?"

Comment author: Yvain 07 April 2011 09:53:52PM *  12 points [-]
  • Data analysis for a really great experiment on calibration by one of my lecturers. He got about 20000 people to do a calibration test and fill in forms about all their demographic data, and now we're going to see what factors affect how well-calibrated people are. Also involved is some attempts to get them to predict political events in the year 2010 and see how ridiculously wrong they were. Currently bumbling along with SPSS to make it yield results and inventing new scoring rules. Timescale: Finish before summer vacation.

  • I've started a FAQ on why everyone should be consequentialists similar to my Non-Libertarian FAQ. People say they're not consequentialists all the time, and it really annoys me, and I need a document to link them to. This has proven harder than I expected, mostly for organizational reasons. I have a feeling a lot of it is going to be gathering some of Eliezer's arguments in one place and making them explicit. Timescale: Sometime in 2011?

  • About halfway through first draft of a sequence on the role of reinforcement in psychology. This may require endless revisions if it's going to be done right. Timescale: Hofstadteresque.

  • Studying for my second step of the United States Medical Licensing Exam this summer.

  • On the just-for-fun side, slowly learning some more ins and outs of Bryce, Sketchup, and other 3D graphics programs. My latest interest is mapping fictional cities.

  • Occasional short stories when the inspiration strikes.

  • And I'm finished with my Dungeons and Discourse roleplaying system, now working on a campaign for it. I might ask around for players sometime.

Comment author: Emile 03 April 2011 08:21:05AM 0 points [-]

I'm fine with rescheduling, I didn't get a lot of responses beyond "I'm on holiday", "I live too far away" and "maybe" ... so would you prefer the weekend of April 23rd or April 30?

Comment author: Yvain 06 April 2011 09:33:08PM 0 points [-]

I've finalized my plans and will be there from 27/4 to 2/5 - so the weekend of the 30th sounds good to me.

Comment author: Risto_Saarelma 04 April 2011 01:03:01PM 30 points [-]

My friend, Tony, does prop work in Hollywood. Before he was big and famous, he would sell jewelry and such at Ren Faires and the like. One day I'm there, shooting the shit with him, when a guy comes up and looks at some of the crystals that Tony is selling. he finally zeroes in on one and gets all gaga over the bit of quartz. He informs Tony that he's never seen such a strong power crystal. Tony tells him it a piece of quartz. The buyer maintains it is an amazing power crystal and demands to know the price. Tony looks him over for a second, then says "If it's just a piece of quartz, it's $15. If it's a power crystal, it's $150. Which is is?" The buyer actually looked a bit sheepish as he said quietly "quartz", gave Tony his money and wandered off. I wonder if he thought he got the better of Tony.

-- genesplicer on Something Awful Forums, via

Comment author: Yvain 05 April 2011 11:36:38PM *  16 points [-]

Story kind of bothers me. Yeah, you can get someone to pretend not to believe something by offering a fiscal reward, but that doesn't prove anything.

If I were a geologist and correctly identified the crystal as the rare and valuable mineral unobtainite which I had been desperately seeking samples of, but Tony stubbornly insisted it was quartz - and if Tony then told me it was $150 if it was unobtainite but $15 if it was quartz - I'd call it quartz too if it meant I could get my sample for cheaper. So what?

Comment author: Yvain 04 April 2011 10:52:02AM *  23 points [-]

I've never been in this situation and I can't imagine what you're going through.

But when I have positions that get challenged by a lot of people, I have had some success in writing very long and complete essays detailing why I hold the position, along with all of the responses I expect to get and why they're wrong, and putting it on a blog or website. Then when someone asks why I believe X, I just tell them I'll send them a link to the essay. It weeds out the people who don't care enough to go to a link, and it lets the people who really want to know see the position defended as best I can without having to come up with it on the fly. If there's any pre-existing explanation of atheism you really identify with, you could use that too.

And I have had good experiences with religious people by confounding as many atheist stereotypes as possible: being exaggeratedly nice and understanding, mentioning how much I enjoy religious music / religious writing / the teachings of Jesus / whatever else I honestly respect about religion but saying that some other parts aren't for me, not bringing the issue up but having a few overwhelmingly strong points that they will agree with when it is brought up, and having a link to a more complete argument ready in case I feel a discussion is getting too confrontational and counterproductive.

I also find that if my goal is just to end an argument without losing too much social capital or coming across as confrontational, I get better results with emotional rather than intellectual points, as long as the emotional points are framed in a nonconfrontational and nonchallenging way. Going on about Biblical contradictions just gets a "You're obviously proud of your worldly learning, but worldly learning leads you astray" or something from the less intellectual, and an attempt to rationalize the contradiction from the more intellectual. But if I say that some of my Jewish relatives died in the Holocaust and I don't accept that a just God would allow that to happen, most people have the social graces not to go into a full-fledged explanation of proposed solutions to the problem of evil and to just let the matter rest, or to say that they think my heart is in the right place and they'll pray for me or something, which is really the best one can expect in these sorts of situations.

Comment author: Giles 02 April 2011 03:51:52PM 5 points [-]

This is slightly off-topic (as it doesn't help distinguish between Yvian's hypothesis and T&C's) but anyway:

People who feel guilty sometimes give to charity, right?

Is the social purpose of giving (in this case) therefore to punish yourself financially rather than actually help anyone?

Comment author: Yvain 03 April 2011 09:12:02PM 1 point [-]

Maybe it's to signal that your misdeed was a one-time lapse rather than coming from any deep inner tendency.

For example, if you run over a cat, you might donate money to an animal shelter, which shows that despite what happened you really like animals and would never run one over intentionally.

The financial punishment angle can't hurt, though.

Comment author: Vaniver 31 March 2011 10:21:45PM 4 points [-]

I suspect that's a feature. Part of the genius of the Ottoman method is the only sons that become Sultans are survivors.

Comment author: Yvain 02 April 2011 11:23:23PM *  4 points [-]

It's good to know you still take your responsibility to defend the Ottomans seriously.

Comment author: Yvain 02 April 2011 10:23:03PM *  0 points [-]

I'm planning to go to Paris the last week in April. I would attend a Paris meetup if things worked out (although I don't speak French so I suppose I'd be the annoying tourist forcing you all to talk in English) If you'd be willing to reschedule, let me know and I'll finalize my plans ASAP and tell you what dates would work.

Comment author: saturn 01 April 2011 11:03:47PM 0 points [-]

On the other hand, without some kind of prompting, many doctors tend to narrowly focus on reaching a firm diagnosis at the expense of maximizing benefit/minimizing risk to the patient.

Comment author: Yvain 02 April 2011 11:13:32AM *  4 points [-]

There are certainly different schools of thought possible on how much time to invest in disease identification before going for a treatment, but can you explain your evidence for why you think most doctors tend to err on the side of over-caution?

Medicine does include the ideas of "empirical treatment" and "empirical diagnosis". Empirical treatment is when eg a doctor can't figure out exactly what a disease is, but it looks bacterial, so ey'll throw some common antibiotics at it and see if it works. Empirical diagnosis is when a doctor isn't sure about a particular diagnosis, so ey gives the treatment for that diagnosis and then if the patient gets a bit better ey's convinced and moves on to more serious long-term treatment.

These are useful but still not as good as knowing what you're doing. First of all, a lot of the time they complicate the picture enough to make real diagnosis impossible; an organism that could have been cultured and identified before starting empirical antibiotics might be decimated enough to become unidentifiable afterwards, but not so dead that it can't bounce back and cause a relapse, so starting empirical treatment can be dangerous if you think you're ever going to change your mind and want to figure out exactly what's wrong.

And a lot of time, the treatment for one disease will make another similar disease worse. Steroids are the best treatment for a lot of auto-immune conditions, but they will make infectious conditions worse; immune conditions can look like infectious conditions if you haven't investigated them properly; this is especially true in rashes, which some doctors prescribe steroids for almost automatically. With rashes it's not the end of the world because even if a rash gets worse you probably won't die from it, but it still annoys patients and there are some other conditions where you don't have that margin of error.

There's also just the common sense problem of people getting really angry if they've been told they have a disease, they've changed their life around to help live with that disease, they've been taking medicine for that disease which may have all sorts of side effects, and then they go to a second doctor and that doctor tells them they don't in fact have the disease. I saw a lady the other day who for eleven years thought she had multiple sclerosis, switched neurologists, and then the new neurologist told her she didn't have it - that's an extreme and very rare example and I have a feeling the error is with the new guy rather than the old guy, but even if so that level of uncertainty and confusion is still traumatic and is a good reason for doctors to work really hard to get a firm diagnosis before giving it.

Maybe I'm missing your point; if so can you give an example of what you mean?

Comment author: Bo102010 01 April 2011 05:43:50PM 2 points [-]

Would a patient thus do better to research his or her symptoms online before going to the doctor's office, and then insisting on the treatment provided?

If so is there a good place to do this research? Are there good websites that are usually informed with up-to-date research on a variety of topics? I haven't had a health issue that's needed such research in recent memory, but if I did I would probably type my symptoms into Google alongside technical sounding words like "incidence," "epidemiology," and "differential diagnosis."

Comment author: Yvain 01 April 2011 09:27:14PM *  7 points [-]

Symptom checker programs and Google are good preliminary resources, especially if you want to figure out whether it's worth your time going to the doctor or not for something mild.

If you Google your symptoms, then go to the doctor and insist you have whatever disease came up on top, and tell the doctor ey's wrong if ey disagrees or wants to do further tests, then you become the kind of patient who doctors tell hilarious stories about in hospital cafeterias. These stories are rarely hilarious for the patient involved unless they have a very masochistic sense of humor.

The more honest symptom checkers will give you a list of many possible conditions that could be causing your symptoms. Unless you have something really obvious, most doctors won't be able to do better than a list until they've done tests. Symptom checkers usually don't adjust for your age, your past history, your current medications, or many other things that are very important for serious illnesses, and they can only give a very probabilistic result. They have their place but in a serious condition are no replacement for a doctor.

Comment author: wedrifid 01 April 2011 08:04:50AM 7 points [-]

Here is what Eddy has found in his research. Give a group of cardiologists high-quality coronary angiograms (a type of radiograph or x-ray) of typical patients and they will disagree about the diagnosis for about half of the patients. They will disagree with themselves on two successive readings of the same angiograms up to one-third of the time. Ask a group of experts to estimate the effect of colon-cancer screening on colon-cancer mortality and answers will range from five percent to 95 percent.

An excerpt. Scary!

Comment author: Yvain 01 April 2011 09:17:00PM *  10 points [-]

Unfortunately, Scientific American doesn't give any clues as to what study this information comes from, and my attempts to find it have all come up blank (anyone else have more luck?)

Of the two studies I can find on inter-rater reliability of coronary angiography, one shows greater than 98%, and the other shows high 90s. No doubt these are completely different types of coronary scans with completely different criteria for success than whatever Eddy was doing, but given the lack of pointers to the original study it's impossible to say exactly what's going on.

Comment author: Yvain 01 April 2011 11:32:24AM *  15 points [-]

Well, the good news is that between Robin Hanson's claim that we receive twice as much health care as we should and these guys' claim that "Americans only receive about half of recommended medical care", on average we're getting it pretty darned close!

Comment author: djcb 31 March 2011 02:55:07PM *  2 points [-]

I find evolutionary-psychological reasoning always a bit suspect because it seems a bit too good to explain just about anything. Having said that -- sometimes dogs seem to have a sense of guilt -- does that imply they have a kind of morality too(*)? Or is it just some kind of 'act' due to co-evolution with humans?

(*) De Waal in Primates and Philosophers argues that some animals have a certain degree of morality (Robert Wright slightly disagrees with De Waal, in the same book).

Comment author: Yvain 31 March 2011 07:33:48PM *  7 points [-]

I sort of agree with this; it's temptingly easy to explain away complex higher-level behaviors as adaptations.

But unless I am missing something, the alternative to an evolutionary theory of guilt isn't a non-evolutionary theory of guilt. It's to say "Guilt? Well, obviously if you do something immoral, then you feel bad afterwards, because on some level you know it was wrong." As far as I know once you've reached the point where you really feel like emotions require explanations and you understand that the explanations cannot themselves be mental, evolutionary psychology is pretty much the only game in town.

I'm not saying the particular formulation of guilt presented here is right - maybe it's Tooby and Cosmides' model, maybe it's something else no one's ever thought of, but I think any accurate model of guilt would sound just as reductionist as this one.

Guilt in dogs seems to be mostly illusory.

Comment author: rwallace 31 March 2011 10:55:12AM 6 points [-]

But, even if we can't ethically test it as a prediction, we can at least try to test it as a postdiction. Consider people who've committed atrocities ranging from gang violence to the Holocaust, such that punishment comes late and unexpectedly. From the accounts I've read, it seems to me the following are both true and what we would expect based on this theory:

  1. While the perpetrators are riding high, in their circle of fellow perpetrators, subject to no disapproval and apparently safe from punishment, they don't feel particularly guilty.

  2. Later, when it all comes crashing down and they're in the power of a wider society that despises them and is likely to punish them for their crimes, they often feel - not just consciously fake, but genuinely feel - a great deal of guilt.

Comment author: Yvain 31 March 2011 11:22:20AM 5 points [-]

That sounds more like an argument in favor of Tooby and Cosmides' theory; at the very least it doesn't seem kosher that I used the opposite as support for my theory in the paragraph starting "guilt is just as associated with good outcomes as bad outcomes". Maybe I didn't think it through enough.

Comment author: Yvain 31 March 2011 10:26:13AM *  32 points [-]

I say this just to offer evidence that something about "rationality" works.

Rationality working is one possible explanation of this, but it's not the only one or even the most likely.

There are all sorts of interesting sociological differences between actively religious people and the nonreligious, usually to the advantage of theists. They live longer, report greater happiness, are healthier by most measures of health, and I think have some protection against mental disease. Most studies investigating these advantages find they have nothing to do with the content of the religion and everything to do with the religion providing easy access to the religious community, a friendly and supportive social group to which other believers have an automatic "in".

I have a feeling this works in more subtle ways than just the obvious; it's not just about going to church and seeing people, but about slowly absorbing these people's norms (which are usually pretty positive in practice even when the theory behind them is repulsive) and internalizing their conception of you as a genuinely okay person because you're part of the in-group.

A lot of what you're talking about sounds potentially mediated by the same factors. You are part of a large and active RL community of rationalists and may have internalized the idea of fellow rationalists as your in-group, which means you're adjusting your behavior to conform to rationalist norms and values rather than the norms and values of whatever was your in-group before.

This is not to devalue the importance of the material - most of us would not fit into a religious community no matter how hard we tried and so the material deserves a lot of credit as the attractor around which a community of interesting non-religious people can form - but I think the value of the material is indirect rather than direct.

Guilt: Another Gift Nobody Wants

67 Post author: Yvain 31 March 2011 12:27AM

Evolutionary psychology has made impressive progress in understanding the origins of morality. Along with the many posts about these origins on Less Wrong I recommend Robert Wright's The Moral Animal for an excellent introduction to the subject.

Guilt does not naturally fall out of these explanations. One can imagine a mind design that although often behaving morally for the same reasons we do, sometimes decides a selfish approach is best and pursues that approach without compunction. In fact, this design would have advantages; it would remove a potentially crippling psychological burden, prevent loss of status from admission of wrongdoing, and allow more rational calculation of when moral actions are or are not advantageous. So why guilt?

In one of the few existing writings I could find on the subject, Tooby and Cosmides theorize that "guilt functions as an emotion mode specialized for recalibration of regulatory variables that control trade-offs in welfare between self and other."

If I understand their meaning, they are saying that when an action results in a bad outcome, guilt is a byproduct of updating your mental processes so that it doesn't happen again. In their example, if you don't share food with your sister, and your sister starves and becomes sick, your brain gives you a strong burst of negative emotion around the event so that you reconsider your decision not to share. It is generally a bad idea to disagree with Tooby and Cosmides, but this explanation doesn't satisfy me for several reasons.

First, guilt is just as associated with good outcomes as bad outcomes. If I kill my brother so I can inherit the throne, then even if everything goes according to plan and I become king, I may still feel guilt. But why should I recalibrate here? My original assumptions - that fratricide would be easy and useful - were entirely correct. But I am still likely to feel bad about it. In fact, some criminals report feeling "relieved" when caught, as if a negative outcome decreased their feelings of guilt instead of exacerbating them.

Second, guilt is not only an emotion, but an entire complex of behaviors. Our modern word self-flagellation comes from the old practice of literally whipping one's self out of feelings of guilt or unworthiness. We may not literally self-flagellate anymore, but when I feel guilty I am less likely to do activities I enjoy and more likely to deliberately make myself miserable.

Third, although guilt can be very private it has an undeniable social aspect. People have messaged me at 3 AM in the morning just to tell me how guilty they feel about something they did to someone I've never met; this sort of outpouring of emotion can even be therapeutic. The aforementioned self-flagellators would parade around town in their sackcloth and ashes, just in case anyone didn't know how guilty they felt. And we expect guilt in certain situations: a criminal who feels guilty about what ey has done may get a shorter sentence.

Fourth, guilt sometimes occurs even when a person has done nothing wrong. People who through no fault of their own are associated with disasters can nevertheless report "survivor's guilt" and feel like events were partly their fault. If this is a tool for recalibrating choices, it is a very bad one. This is not a knockdown argument - a lot of mental adaptations are very bad at what they do - but it should at least raise suspicion that there is another part to the puzzle besides recalibration.


THE PARABLE OF THE LAWYER

Suppose you need a lawyer for some important and very lucrative legal case. And suppose by a freak legislative oversight, your state has no laws against legal malpractice and unethical lawyers can get off scot-free. You are going to want to invest a lot of effort into evaluating the morals of the many lawyers anxious to take your case.

One lawyer you meet, Mr. Dewey, has an unusual appearance. A small angel, about the size of a rat, sits on his right shoulder holding an electric cattle prod. This is remarkable, and so you remark upon it.

Mr. Dewey scowls. "That angel has been sitting there for as long as I can remember," he tells you. "Every time I do something wrong, she pokes me with her prod. If it's a minor sin like profanity, maybe she'll only poke me once or twice, but if I lie or swindle, she'll turn the power up on max and keep shocking me for days. It's a miserable, miserable existence, and I'm constantly scared to death I'll slip up and make her angry, but I can't figure out how to get rid of her."

You express some skepticism about this story, so Mr. Dewey offers to demonstrate. He says a mild curse word, and sure enough, the angel pokes him with the cattle prod, giving him a mild electric shock.

Suddenly, Mr. Dewey is a very attractive candidate for your lucrative case. You can be assured that he won't swindle you, because whatever gains he might take from the swindle are less attractive than the punishment he would get from the angel afterwards.

Surgeon Paul Brand considered pain so useful to the body's functioning that he called it "the gift nobody wants". Mr. Dewey's angel is also such a gift, even though he might not appreciate it: clients worried about ethical issues will bring their patronage to his law firm, giving him a major advantage over the competition.

Whereas normally we must trust a lawyer's altruism if we expect em not to con us, in Mr. Dewey's case we need only trust him to pursue his own self-interest. This, then, is the role of guilt: it provides assurance to others that we will be punished for our misdeeds even if there is no external authority to punish us, avoiding Parfitian hitchhiker  dilemmas and ensuring fair play. The assurance of punishment ensures fair play and makes mutually beneficial transactions possible.

FAKEABLE AND UNFAKEABLE SIGNALS

The big difference between Mr. Dewey and ourselves is that where Mr. Dewey has unquestionable evidence of his commitment to self punishment in the form of a very visible angel on his shoulder, for the rest of us guilt is a private mental affair and can be faked. It would seem to be a winning strategy, then, to claim a tendency to guilt while not really having one.

Ms. Wolfram is Mr. Dewey's main competitor, and is outraged at her rival's business success. In an attempt to even the scales, she buys a plastic angel figure from the local church and glues it to her shoulder. "Look!" she tells clients. "I, too, suffer pain when I commit misdeeds!" Her business shoots up to the same high levels as Mr. Dewey's.

One day, the news comes that Mr. Dewey was spotted whipping himself in the town square. When asked why, he explained that in a moment of weakness, he had overcharged a customer. His angel, who had lost its cattle prod, was mind-controlling him into the self-flagellation in place of its more usual punishment.

This provides an impressive bar for Ms. Wolfram to live up to. Sure, she could just whip herself like Mr. Dewey is doing. But it wouldn't be worth it - she just doesn't like the money enough that she would whip herself after every swindle just to drum up business. If she's going to have to whip herself to fake remorse whenever she commits wrongdoing,  her best policy really is to genuinely stop swindling people.

Mr. Dewey has found an unfakeable signal. Even though whipping himself in public is one of the most unpleasant things he could do, in this case it is good business practice. It once again differentiates him from Ms. Wolfram and restores his status as the city's most desirable attorney.

In evolutionary terms, guilt becomes more credible the more it requires publicly visible behavior that no reasonable cheat would want to fake. Hurting oneself, avoiding pleasurable activities, lowering your own status, and withdrawing from social activities are all evolutionary costly and therefore good ways to prove you are experiencing guilt; the usual vocal, postural, and facial cues of being miserable are also useful.

There's no reason people should evolve an all-consuming sense of guilt. If an opportunity comes along where the benefits of cheating are greater than the social costs, an organism should still take it. Therefore, guilt has to be unpleasant but not infinitely unpleasant. A person who committed suicide in response to even the slightest moral infraction would be trustworthy, but they'd miss out if an excellent opportunity to win major gains for cheating happened to fall into their lap.

The conspicuous experience of guilt is an evolutionarily advantageous way of assuring potential trading partners that you will be punished for defection. The behaviors associated with guilt are costly signals that help differentiate false claims of guilt from the real thing and add to public verifiability of the punishment involved.

UNDESERVED GUILT

If you kill your brother in order to inherit the throne, you probably deserve whatever guilt you feel. But in the phenomenon of "survivor's guilt", people feel guilt for events that weren't even remotely their fault. Maybe you go hiking with your brother, and through no fault of your own he trips and falls down a crevasse and dies, and now you feel guilty. Why?

Hunter-gatherer societies were more violent than our own; statistics differ but by some estimates around 30% of hunter-gatherer males died of homicide. Even as late as the Bronze Age, Biblical figures who killed their brothers comprise a rather impressive list including Cain, Solomon, Ammon, Abimelech, and Jehoram; Jacob's sons merely attempted to do so. So the priors for suspicious death must have been very different in the olden days.

Further, in such a crime-ridden culture, there may have been more incentives to blame an enemy for a death, even if that enemy was not responsible. A person whose brother has accidentally died on a hiking trip with no witnesses would be very targetable.

And even in less drastic situations than blaming survivors for a death, there may be other possible threats to reputation. If there is only one survivor of a battle, he may be suspected of cowardice; if there is only one survivor of a disaster, she may be suspected of running away without helping others.

Therefore, it would be advantageous to have a method of proving your innocence. Suppose that you would gain benefits X from killing your brother and covering it up, but that you would suffer losses Y if you were suspected of the crime and punished. A precommitment to a policy of experiencing a level of guilt between X and Y provides a tool for proving your innocence. It would no longer be in your self-interest to kill your brother, because you will suffer so much guilt that you won't be able to enjoy the benefits of your crime; your would-be accusers realize this and admit your innocence, saving you from the still worse outcome Y.

In this case, guilt would be an entirely adaptive response to a disaster with which you were associated, even if your own actions were beyond reproach. A level of unhappiness worse than any benefits you could get by profiting the tragedy, but less than any punishment you might receive if you were suspected of profiting from the tragedy, would be helpful in clearing your name of any wrongdoing.

(The proposed mechanism is almost identical to one cited in Thornhill and Palmer's controversial and unpleasant evolutionary account of post-traumatic stress after rape.)

This theory makes some testable predictions, which as far as I know have not been tested:

- People should feel guiltier about events for which reasonable suspicion might exist that they played a part; for example, if your brother slipped and fell while you were hiking alone with him rather than in a large group with many witnesses.
- People should feel guiltier about events for which they might profit; for example, if you stood to inherit money from your brother, or never liked him much anyway.
- People may be suspicious of people who come out of a disaster feeling no survivor's guilt.

CONCLUSION

Guilt, like pain, is "a gift nobody wants". Because people with guilt are known to punish themselves for moral wrongdoing, their social group considers them more trustworthy and they gain the advantages of trade and cooperation. In order to prove that their guilt is real rather than feigned, they use costly signals like deliberate self-harm and self-denial to display their punishment publicly

When one has done nothing wrong, it can sometimes be advantageous to paradoxically display guilt in order to prove one's lack of wrongdoing. These costly signals demonstrate that it is not in one's self-interest to lie about these matters, while still being less costly than the punishment for defection.

Although this could theoretically be mediated by the behavioral strategies of a sufficiently intelligent and Machiavellian unconscious mind, it fits within the framework of evolutionary psychology and can also be interpreted in evolutionary terms.

In response to Faith and theory
Comment author: Yvain 27 March 2011 12:33:43AM 34 points [-]

I think you lost me when you assumed "faith" ought to be a meaningful word with a coherent definition.

I think the best definition to give for faith is a practical one: faith is the word people use as a combination semantic stop-sign and applause light when asked why they believe in religion. If someone then goes all philosophical on them and asks them what exactly they mean, they then use whatever plausible explanation seems appropriate.

I've heard faith described as:

  • exactly the same as inductive reasoning; thus, you have "faith" that the sun will rise tomorrow.
  • similar to the concept of credibility; the Bible's always been right before, so I imagine it will be right on this one issue here, even though I'm not exactly sure how.
  • similar to the concept of trust: God is a good being, I'm sure He knows what He's doing.
  • a direct meddling of the Holy Spirit in the internal workings of your brain, so that you are able to mystically come to the right answer about whether or not to believe in religion despite insufficient evidence.
  • a decision to follow whatever idea seems intuitively true or most pleasant, and to give evidence less weight than these intuitions
  • a decision to believe something even though you have no reason for doing so; also the thing those people who ask the obvious followup question "but why did you choose to believe that for no reason, as opposed to something else for no reason" should feel bad for not having.
  • Not comprehensible to the limited human mind so don't talk about it

I don't think any of these definitions are "the correct definition"; I just think they're different ways that people in different situations and with different degrees of philosophicalness cash out the idea of "I believe in religion and you can't tell me not to and I feel pretty good about it"

As such, I don't believe there's a concept called "faith" which it is necessary to distinguish from theory in the first place.

Comment author: lukeprog 23 March 2011 12:15:16PM *  4 points [-]

Eliezer and I, over the course of our long discussion, have come to some understanding of what would constitute useful. Though, Philosophy_Tutor suggested that Eliezer taboo his sense of "useful" before trying to declare every item on my list as useless.

Whether or not I can provide a set of necessary and sufficient conditions for "useful", I've repeatedly pointed out that:

  1. Several works from mainstream philosophy do the same things he has spent a great deal of time doing and advocating on Less Wrong, so if he thinks those works are useless then it would appear he thinks much of what he has done on Less Wrong is uesless.

  2. Quite a few works from mainstream philosophy have been used by him, so presumably he finds them useful.

I can't believe how difficult it is to convince some people that some useful things come out of mainstream philosophy. To me, it's a trivial point. Those resisting this truth keep trying to change the subject and make it about how philosophy is a diseased subject (agreed!), how we shouldn't read Quine (agreed!), how other subjects are more important and useful (agreed!), and so on.

Comment author: Yvain 23 March 2011 06:54:24PM 12 points [-]

I can't believe how difficult it is to convince some people that some useful things come out of mainstream philosophy. To me, it's a trivial point.

If it's not immediately obvious how an argument connects to a specific implementable policy or empirical fact, default is to covertly interpret it as being about status.

Since there are both good and bad things about philosophy, we can choose to emphasize the good (which accords philosophers and those who read them higher status) or emphasize the bad (which accords people who do their own work and ignore mainstream philosophy higher status).

If there are no consequences to this choice, it's more pleasant to dwell upon the bad: after all, the worse mainstream philosophy does, the more useful and original this makes our community; the better mainstream philosophy does, the more it suggests our community is a relatively minor phenomenon within a broader movement of other people with more resources and prestige than ourselves (and the more those of us whose time is worth less than Eliezer's should be reading philosophy journals instead of doing something less mind-numbing).

I think this community is smart enough to avoid many such biases if given a real question with a truth-value, but given a vague open question like "Yay philosophy - yes or no?" of course we're going to take the side that makes us feel better.

I think the solution is to present specific insights of Quinean philosophy in more depth, which you already seem like you're planning to do.

In response to Admit your ignorance
Comment author: RichardKennaway 15 March 2011 12:38:11PM 5 points [-]

I might lose the respect of a tutor, or even a lecturer by admitting my ignorance

That would be a fault in the tutor or lecturer. No-one that I teach will lose my respect for revealing their ignorance or lack of understanding of something (nor for concealing it either). I listen and observe and try to say whatever I think will address their difficulty, and if it doesn't, try something else. And it sounds as if you act similarly in your teaching.

Comment author: Yvain 15 March 2011 10:27:17PM 5 points [-]

There are also issues around classmates. Even when I don't think I will lose the respect of my classmates for holding up the class until something is explained to my satisfaction, I do expect I will test their patience if it takes me an unusually long time to understand something. Certain concepts I just can't get on a first-pass explanation when put on the spot; it's almost always easier to pretend I understand it and then go home and study it.

Comment author: Yvain 14 March 2011 09:59:13PM *  3 points [-]

Back when I had the same question, I found http://www.antipope.org/charlie/blog-static/2008/02/politics.html most enlightening. Once I realized it wasn't really about making good policy or making sure people were really represented it made much more sense.

In response to comment by Yvain on Being a teacher
Comment author: Swimmer963 14 March 2011 09:35:08PM 1 point [-]

Do you know the rule now why you say 'big brown' and not 'brown big'? I don't...I'm very curious though!

In response to comment by Swimmer963 on Being a teacher
Comment author: Yvain 14 March 2011 09:53:15PM *  3 points [-]

I don't think there's any justification more interesting than "do it this way because that's how it's done", but if it helps, at least it's annoyingly complicated.

In response to Being a teacher
Comment author: Yvain 14 March 2011 09:23:15PM *  30 points [-]

I used to teach English as a second language. It was a mind trip.

I remember one of my students saying something like "I saw a brown big spider". I responded "No, it should be 'big brown spider'". He asked why. Not only did I not know the rule involved, I had never even imagined that anyone would ever say it the other way until that moment.

Such experiences were pretty much daily occurrences.

Comment author: AlephNeil 11 March 2011 11:28:29PM *  0 points [-]

First I want to note that "the sensation of pain, considered in and of itself" is, after "the redness of red", the second most standard example of "qualia". So if, like good Dennettians, we're going to deny that qualia exist then we'd better deny that "the sensation of pain in and of itself" has moral disvalue!

Instead we should be considering: "the sensation of pain in and of how-it-relates-to-other-stuff". So how does pain relate to other stuff? It comes down to the fact that pain is the body's "damage alarm system", whose immediate purpose is to limit the extent of injury by preventing a person from continuing with whatever action was beginning to cause damage.

So if you want to deny qualia while still holding that pain is morally awful (albeit not "in and of itself") then I think you're forced at least some of the way towards my position. 'Pain' is an arrow pointing towards 'damage' but not always succeeding - a bit like how 'sweetness' points towards 'sugar'. This is an oversimplification, but one could almost say that I get the rest of the way by looking at where the arrow is pointing rather than at the arrow itself. (Similarly, what's "bad" is not when the smoke alarm sounds but when the house burns down.)

That position implies we should be indifferent between torturing ... (or if you try to escape by saying all pain causes permanent psychological damage

Well, it's manifestly not true that all pain causes permanent psychological damage (e.g. the pain from exercising hard, from the kind of 'play-fighting' that young boys do, or from spicy food) but it seems plausible that 'torture' does.

then we should still be indifferent between killing a person quickly, or torturing em for six hours and then killing em).

I admit this gave me pause.

There's a horrible true story on the internet about a woman who was lobotomised by an evil psychologist, in such a way that she was left as a 'zombie' afterwards (no, not the philosophers' kind of zombie). I've rot13ed a phrase that will help you google it, if you really want, but please don't feel obliged to: wbhearl vagb znqarff.

Let it be granted that this woman felt no pain during her 'operation' and that she wasn't told or was too confused to figure out what exactly was happening, or why the doctor kept asking her simple things - e.g. her name, or to hum her favourite tune. (The real reason, as best I can tell, was "to see how well a person with that amount of brain damage could still do so and so")

What I want to say is that even with these elaborations, the story is just as repulsive - it offends our moral sense just as much - as any story of torture. This would still be true even if the victim had died shortly after the lobotomy. More to the point, this story is massively more repulsive than, say, a story where Paul Atreides has his hand thrust into a 'pain box' for half an hour before being executed (presumably by a gom jabbar). (And that in turn feels somewhat more repulsive than a story where Paul Atreides is tricked into voluntarily holding his hand in the pain box and then executed, despite the fact that the pain is just as bad.)

Torture isn't just a synonym for "excruciating pain" - it's more complicated than that.

Here's a more appetising bullet: We should be indifferent between "Paul Atreides executed at time 0" and "Paul Atreides tricked into voluntarily holding his hand in the pain box for half an hour and then executed". (I'd be lying if I said I was 100% happy about biting it, but neither am I 100% sure that my position is inconsistent otherwise.)

Comment author: Yvain 12 March 2011 12:56:30AM 3 points [-]

First I want to note that "the sensation of pain, considered in and of itself" is, after "the redness of red", the second most standard example of "qualia". So if, like good Dennettians, we're going to deny that qualia exist then we'd better deny that "the sensation of pain in and of itself" has moral disvalue!

I know Dennett's usually right about this sort of thing and so there must be something to that argument, but I've never been able to understand it no matter how hard I try. It looks too much like wishful thinking - "these qualia things are really confusing, so screw that." Certainly it's not the sort of Reduction with a capital "R" I've heard that's left me genuinely satisfied about the nonexistence of things like Free Will or Good or Essence-Of-Chair-Ness.

I would be hesitant to say the sensation of pain in and of itself has moral disvalue; I would say that people have preferences against pain and that the violation of those preferences causes moral disvalue in the same sense as the violation of any other preference. I would have no trouble with inflicting pain on a masochist, a person with pain asymbolia, or a person voluntarily undergoing some kind of conditioning.

Damage can also be something people have a preference against, but it's not necessarily more important than pain. There are amounts of torture such that I would prefer permanently losing a finger to undergoing that torture, and I suspect it's the same for most other people.

What I want to say is that even with these elaborations, the story is just as repulsive - it offends our moral sense just as much - as any story of torture. This would still be true even if the victim had died shortly after the lobotomy.

You seem to be arguing from "Things other than pain are bad" to "Pain is not bad", which is not valid.

I admit your Paul Atreides example doesn't disgust me so much, but I think that's because number one I have no mental imagery associated with gom jabbars, and number two I feel like he's a legendary Messiah figure so he should be able to take it.

If we start talking about non-Kwisatz Haderach people, like say your little sister, and we start talking about them being whipped to death instead of an invisible and inscrutable gom jabbar, I find my intuition shifts pretty far the other direction.

I'd be lying if I said I was 100% happy about biting it, but neither am I 100% sure that my position is inconsistent otherwise.

So I'm reading about your moral system in your other post, and I don't want to get into debating it fully here. But surely you can recognize that just as some things and systems are beautiful and fascinating and complex, there are other systems that are especially and uniquely horrible, and that it is a moral credit to remove them from the world. Sometimes I read about the more horrible atrocities perpetrated in the Nazi camps and North Korea, and I feel physically sick that there is no way I can't just kill everyone involved, the torturers and victims both, and relieve them of their suffering, and that this is the strongest moral imperative imaginable, much more important than the part where we make sure there are lots of rainforests and interesting buildings and such. Have you never felt this emotion? And if so, have you ever read a really good fictional dystopian work?

Comment author: Morendil 11 March 2011 04:13:52PM 3 points [-]

FWIW I'm basically in the same position as AlephNeil's (and I'm puzzled at the two downvotes: the response is informative, in good faith, and not incoherent).

If you (say, you-on-the-other-side-of-the-camera-link) hurt me, the most important effects from that pain are on my plans and desires: the pain will cause me to do (or avoid doing) certain things in the future, and I might have preferred otherwise. Maybe I'll flinch when I come into contact with you again, or with people who look like you, and I would have preferred to look happy and trusting.

It's not clear that the ECPs as posited are "feeling" in pain in the same sense; if I refrain from pushing the button, so that I can pocket the $100, I have no reason to believe that this will cause the ECP to do (or avoid doing) some things in future when it would have preferred otherwise, or cause it to feel ill-disposed toward me.

As for pleasure, I think pleasure you have not chosen and that has the same effect on you as a pain would have (derailing your current plans and desires) also has moral disvalue; only freely chosen pleasure, that reinforces things you already value or want to value, is a true benefit. (For a fictional example of "inflicted pleasure" consider Larry Niven's tasp weapon.)

Comment author: Yvain 11 March 2011 08:22:59PM *  6 points [-]

That position implies we should be indifferent between torturing a person for six hours in a way that leaves no permanent damage, or just making them sit in a room unable to engage in their normal activities for six hours (or if you try to escape by saying all pain causes permanent psychological damage, then we should still be indifferent between killing a person quickly, or torturing em for six hours and then killing em).

I think this is a controversial enough position that even if you're willing to bite that bullet, before you state it you should at least say you understand the implication, are willing to bite the bullet, and maybe provide a brief explanation of why.

Comment author: CharlesR 10 March 2011 08:17:43AM 1 point [-]

Don't forget to consider the weather.

Comment author: Yvain 10 March 2011 12:56:52PM *  2 points [-]

Just make sure you realize you're considering it.

Comment author: CuSithBell 09 March 2011 07:18:07PM 0 points [-]

I agree (with this question) - what makes us so sure that "maximize paperclips" is the part of the utility function that the optimizer will really value? Couldn't it symmetrically decide that "maximize paperclips" is a constraint on "try not to murder everyone"?

Comment author: Yvain 09 March 2011 08:03:50PM *  5 points [-]

Asking what it really values is anthropomorphic. It's not coming up with loopholes around the "don't murder" people constraint because it doesn't really value it, or because the paperclip part is its "real" motive.

It will probably come up with loopholes around the "maximize paperclips" constraint too - for example, if "paperclip" is defined by something paperclip-shaped, it will probably create atomic-scale nanoclips because these are easier to build than full-scale human-sized ones, much to the annoyance of the office-supply company that built it.

But paperclips are pretty simple. Add a few extra constraints and you can probably specify "paperclip" to a degree that makes them useful for office supplies.

Human values are really complex. "Don't murder" doesn't capture human values at all - if Clippy encases us in carbonite so that we're still technically alive but not around to interfere with paperclip production, ve has fulfilled the "don't murder" imperative, but we would count this as a fail. This is not Clippy's "fault" for deliberately trying to "get around" the anti-murder constraint, it's our "fault" for telling ver "don't murder" when we really meant "don't do anything bad".

Building a genuine "respect" and "love" for the "don't murder" constraint in Clippy wouldn't help an iota against the carbonite scenario, because that's not murder and we forgot to tell ver there should be a constraint against that too.

So you might ask: okay, but surely there are a finite number of constraints that capture what we want. Just build an AI with a thousand or ten thousand constraints, "don't murder", "don't encase people in carbonite", "don't eat puppies", etc., make sure the list is exhaustive and that'll do it.

The first objection is that we might miss something. If the ancient Romans had made such a list, they might have forgotten "Don't release damaging radiation that gives us cancer." They certainly would have missed "Don't enslave people", because they were still enslaving people themselves - but this would mean it would be impossible to update the Roman AI for moral progress a few centuries down the line.

The second objection is that human morality isn't just a system of constraints. Even if we could tell Clippy "Limit your activities to the Andromeda Galaxy and send us the finished clips" (which I think would still be dangerous), any more interesting AI that is going to interact with and help humans needs to realize that sometimes it is okay to engage in prohibited actions if they serve greater goals (for example, it can disable a crazed gunman to prevent a massacre, even though disabling people is usually verboten).

So to actually capture all possible constraints, and to capture the situations in which those constraints can and can't be relaxed, we need to program all human values in. In that case we can just tell Clippy "Make paperclips in a way that doesn't cause what we would classify as a horrifying catastrophe" and ve'll say "Okay!" and not give us any trouble.

Comment author: XiXiDu 09 March 2011 12:30:11PM *  0 points [-]

Because an AI built as a utility-maximizer will consider any rules restricting its ability to maximize its utility as obstacles to be overcome. If an AI is sufficiently smart, it will figure out a way to overcome those obstacles. If an AI is superintelligent, it will figure out ways to overcome those obstacles which humans cannot predict even in theory and so cannot prevent even with multiple well-phrased fail-safes.

I hope AGI's will be equipped with as many fail-safes as your argument rests on assumptions.

A paperclip maximizer with a built-in rule "Only create 10,000 paperclips per day" will still want to maximize paperclips. It can do this by deleting the offending fail-safe, or by creating other paperclip maximizers without the fail-safe, or by creating giant paperclips which break up into millions of smaller paperclips of their own accord, or by connecting the Earth to a giant motor which spins it at near-light speed and changes the length of a day to a fraction of a second.

I just don't see how one could be sophisticated enough to create a properly designed AGI capable of explosive recursive self-improvement and yet fail drastically on its scope boundaries.

Unless you feel confident you can think of every way it will get around the rule and block it off, and think of every way it could get around those rules and block them off, and so on ad infinitum, the best thing to do is to build the AI so it doesn't want to break the rules...

What is the difference between "a rule" and "what it wants". You seem to assume that it cares to follow a rule to maximize a reward number but doesn't care to follow another rule that tells it to hold.

Comment author: Yvain 09 March 2011 06:49:17PM *  4 points [-]

What is the difference between "a rule" and "what it wants"?

I'm interpreting this as the same question you wrote below as "What is the difference between a constraint and what is optimized?". Dave gave one example but a slightly different metaphor comes to my mind.

Imagine an amoral businessman in a country that takes half his earnings as tax. The businessman wants to maximize money, but has the constraint is that half his earnings get taken as tax. So in order to achieve his goal of maximizing money, the businessman sets up some legally permissible deal with a foreign tax shelter or funnels it to holding corporations or something to avoid taxes. Doing this is the natural result of his money-maximization goal, and satisfies the "pay taxes" constraint..

Contrast this to a second, more patriotic businessman who loved paying taxes because it helped his country, and so didn't bother setting up tax shelters at all.

The first businessman has the motive "maximize money" and the constraint "pay taxes"; the second businessman has the motive "maximize money and pay taxes".

From the viewpoint of the government, the first businessman is an unFriendly agent with a constraint, and the second businessman is a Friendly agent.

Does that help answer your question?

Comment author: Yvain 09 March 2011 02:53:22PM *  2 points [-]

I admit to not really understanding how different deontic logics work. I would hope it's like propositional logic in that you can't generate false statements out of true premises and all you have to do is set up a theorem-prover and let it do its thing.

But that doesn't seem to be true since the article wants to judge each new system the AI develops against some criteria to make sure it doesn't come up with weird moral laws. And if that's the case, and you really don't know what kind of things it's going to come up with, just using similarity or difference from four statements to judge whether they're okay or not doesn't seem very reassuring.

Especially since Wikipedia discusses how innocent-seeming deontic logics can sometimes accidentally prove "it is obligatory to murder" and it doesn't seem like any of the four statements listed as sanity checks would flag that one as a problem.

It may be that I just haven't internalized Moravec's Paradox enough to view a system that at its best can correctly generate statements like "If A->X and X is forbidden, A is forbidden" as an interesting accomplishment. But couldn't you get the same thing by just using a utility-maximizing AI and normal propositional logic?

Comment author: Yvain 08 March 2011 09:41:57PM *  25 points [-]

Because an AI built as a utility-maximizer will consider any rules restricting its ability to maximize its utility as obstacles to be overcome. If an AI is sufficiently smart, it will figure out a way to overcome those obstacles. If an AI is superintelligent, it will figure out ways to overcome those obstacles which humans cannot predict even in theory and so cannot prevent even with multiple well-phrased fail-safes.

A paperclip maximizer with a built-in rule "Only create 10,000 paperclips per day" will still want to maximize paperclips. It can do this by deleting the offending fail-safe, or by creating other paperclip maximizers without the fail-safe, or by creating giant paperclips which break up into millions of smaller paperclips of their own accord, or by connecting the Earth to a giant motor which spins it at near-light speed and changes the length of a day to a fraction of a second.

Unless you feel confident you can think of every way it will get around the rule and block it off, and think of every way it could get around those rules and block them off, and so on ad infinitum, the best thing to do is to build the AI so it doesn't want to break the rules - that is, Friendly AI. That way you have the AI cooperating with you instead of trying to thwart you at every turn.

Related: Hidden Complexity of Wishes

Comment author: Yvain 08 March 2011 06:52:03PM *  7 points [-]

Less Wrong has a FAQ that anyone can edit. I think your first four Measures could be best addressed by getting an account on the Wiki and writing what you think we need.

Your worries about the structure of SIAI sound like the sort of thing worth talking about, but posting them on Less Wrong might not be the best way to go about them due to bystander effect and lots of important SIAI folk not being readers here. If you are really interested in this side of things, consider emailing someone on the organizational side of SIAI (Eliezer is not primarily organizational side and is usually busy; Michael Vassar might be good or at least know who to forward it to) and seeing what they have to say. Justin Shovelain also has a history of being good at explaining this side of things; he has a sequence that I think will get some of this across somewhere in the pipeline.

Comment author: XiXiDu 06 March 2011 04:38:45PM 4 points [-]

I'd love to see some series of posts where someone applies LWism to such stories. LW is too much about refining rationality when the biggest problem is that people don't know how to apply it to real life problems. But since Yvain has basically bashed that idea I lost all hope that it will ever happen. He actually got 26 upvotes for it and I don't even know of a single post that shows how to apply what you can learn on LW to real life problems.

Comment author: Yvain 07 March 2011 10:27:55PM *  6 points [-]

Looking back on that comment I agree it came across as too harsh.

I meant only that this site's comparative advantage is not in teaching object-level information about how to succeed socially, pick up girls/boys, program computers, or lose weight - even if the post with the information is called "Losing Weight The Rationalist Way".

I totally support more posts on the interaction between rationality and real-life problems, like if there is a specific cognitive bias that's preventing people from losing weight, or if the problem of weight loss illuminates some wider issue about how people think and act. Just not "Hey guys, I just read about how if you only eat carrots for a week you can lose thirty pounds! This sounds really rational and high utility! Bayes!"

(I feel the same way about discussion of politics. I realize different people's intuition about where to draw the line on weight-loss and politics posts are in different places, and I know I've gotten flak for crossing that line a few times, and I'm willing to live and let live on the issue up until it becomes really blatant.)

I would welcome an analysis of this claim, but I think it would be most suited to Less Wrong if the analysis focused on the process of analyzing scientific papers, rather than just being about the biology of cyanobacteria. I'd probably still read the cyanobacteria one, I just wouldn't see why it's on LW.

Comment author: CronoDAS 07 March 2011 12:22:12AM *  3 points [-]

When writing essays for English class in high school, I was explicitly forbidden from using the word "I"...

Comment author: Yvain 07 March 2011 12:31:44AM *  16 points [-]

Did you by any chance go to high school in a Randian dystopia?

Comment author: Vladimir_Nesov 06 March 2011 11:52:43PM *  8 points [-]

consider the fact that even I read this stuff, and I have the mathematical ability of a random rock

Seriously, you are too smart to have any trouble in acquiring understanding of mathematics if you made a serious effort. Just read the textbooks starting at an appropriate level. Given that you characterize your skill the way you do, there's probably some low-hanging fruit there for you. (But it's possible that you won't be able to enjoy the process. I know I would know less math if I didn't have something to protect.)

A fact I didn't appreciate before encountering this whole AI-related craze followed by overcomingbias followed by lesswrong is that it's possible to master an arbitrary field of expertise by systematically teaching yourself its skills, even if it's completely dissimilar to all you've ever known.

Comment author: Yvain 07 March 2011 12:28:19AM 4 points [-]

"Learn math" is kind of a broad imperative. I know the math that's common to many different applications, like arithmetic and algebra and a bit of calculus, but after that it becomes so fractured that even when I learn how to solve one specific problem in a specific field, I never encounter that problem or field again.

If there were a specific cause for which I needed math, I would force myself to learn the math relevant to that cause, but just "learn topology, who knows when you might need it?" has never been very convincing to me.

I've studied a little decision theory, since that seems to be the form of math most relevant to Friendly AI, but so far I've found it frustrating and hard to treat with suitable rigor. If anyone wants to recommend an unusually good textbook, preferably online, I suppose I'd take suggestions.

Comment author: Armok_GoB 06 March 2011 06:43:22PM 0 points [-]

This should be a top level post.

Comment author: Yvain 07 March 2011 12:04:25AM *  1 point [-]

I haven't top-levelled this specific line of thinking because I figured it was already addressed in the two Sequence posts mentioned at the bottom. If people who have read the sequences agree that they didn't get this exact implication when they first read them, I'll top-level it.

Comment author: Yvain 06 March 2011 11:31:00PM *  17 points [-]

Re: blood vessels. If by "cracking a blood vessel in your eye" you mean you got a red bloody spot in your eye when you looked in the mirror for a little while, then it went away, and there was no loss of vision or pain or anything like that, then this is a subconjunctival haemmorhage. It's usually caused by minor trauma like sneezing really hard or rubbing your eyes too hard, and it is perfectly normal and I've gotten one myself. As far as I know it is not related at all to the pathological processes of aneurysms in the brain. Sometimes SCHs can be caused by hypertension which is generally bad, but you can easily check your BP with a BP cuff (there are usually free ones floating around pharmacies and places like that) and even if your BP was mildly elevated it wouldn't be anything a third of the country doesn't also have. If your blood pressure is below 140/90 there is no great medical evidence for bothering too hard to bring it even lower (unless you have diabetes or something like that)..

Re: insects - look up Jainism. They're a religion one of whose tenets is that you can't hurt anyone in any way, including insects, and they have developed a lot of methods for avoiding accidental harm. If your utility function really includes a term for this, drawing off the Jains' two millennia of expertise is your best bet.

As a more general solution to your problem, I would suggest reading the Sequences. If you have to, stop reading new LW posts and just read the Sequences. There is no reason not to read the Sequences. The Sequences are your friends. Everyone loves the Sequences. Do not taunt the Sequences.

...seriously, most new LW posts are either advanced extensions of Sequence material, or fluff. The Sequences are where you should really go if you're looking for a foundation for using probability in your life. Reading the Sequences looks daunting by sheer word count, but it's not like trying to read a calculus textbook. The Sequences are some of the most engaging, enjoyable things I have ever read. I think I finished them all within two weeks of finding the blog (to be fair, there were fewer of them at that point) and when I finished, I lay down and wept that there were no more sequences to read (not really). They're that good. People who keep complaining about having to read the Sequences don't realize how lucky they are that they have the opportunity to read them for the first time and get that much low-hanging fruit in a single go. They are that good.

Eliezer's Intro to Bayes Theorem and (especially) his Technical Explanation of Technical Explanation should really be counted as part of the Sequences for your purposes. If it's math you're dreading, consider the fact that even I read this stuff, and I have the mathematical ability of a random rock. All the math in the Sequences and yudkowsky.net can be skimmed once you have a good idea of the concepts behind it (someone will yell at me here and say no it can't, but I think people who are good at math will underestimate the ability of people who are not good at math to conceptualize the math to the point where they don't need every single equation -- as long as they honestly try to do this and don't just pretend it doesn't exist). Or if math is really the only thing preventing you from reading the Sequences, go ahead and pretend it doesn't exist and you'll still get a treasure trove out of it.

Comment author: Yvain 05 March 2011 08:03:34PM *  46 points [-]

Seeing this makes me happy because I had a similar revelation a few years ago and it always makes me mad to see people use the glaringly bad justification for being pro-choice which you've overcome. On the other hand, after thinking about the matter quite a bit I still am pro-choice. You say:

On the other hand, as little as it is, it still represents a human life

I think the key word is "represents".

A lot of bad reasoning seems to come from proving a controversial idea can be fit into a category of things that are mostly bad, and then concluding that the controversial idea, too, must be mostly bad.

For example, some people are opposed to a project to genetically engineer diseases like cystic fibrosis out of the human genome, because that's a form of "eugenics". I think this is supposed to cash out as saying that the CF project shares some surface features with what the Nazis did and what those American Southerners who tried to force-sterilize black people did, and those two things are definitely bad, so the CF project must also be bad.

The counterargument is that the features it shares with the Nazi project and the Southern project are not the features that made those two programs bad. Those two programs were bad because they involved hurting people, either through death or through force-sterilization, without their consent. The CF elimination project hopefully would be voluntary and would not damage the people involved. Therefore, although it shares some similarities with the Nazi project and the Southern project (it's about genetics, it's intended to improve the species, etc), those aren't relevant to this moral question and the argument "But it's eugenics" is flawed.

(If you haven't read the 37 Ways Words Can Be Wrong sequence, I suggest that now. Think of a person taking a blue egg that contains vanadium, pointing to a bin full of blue eggs that contain palladium, and saying "But this is a blegg, and we all know bleggs contain palladium!" Well, no.)

The "human life" issue strikes me as very similar. "Taking a human life" is a large category mostly full of bad things. It contains things like stabbing a teenager with a knife, poisoning a senator, strangling an old person in a nursing home, starving a toddler, et cetera. All of these are really bad. They're really bad for various reasons including that they cause the person pain, that they disrupt society, that they violate the person's preference not to be killed, et cetera.

Abortion possibly does fit into the category of "taking a human life." But although it shares the surface features of that category, it isn't clear whether or not it shares the interesting moral feature which is exactly what the whole argument is about. Killing you or me is bad because we understand death and have preferences against it and don't want to die. Whether or not killing a fetus is bad depends on whether or not the fetus also satisfies those conditions - not on whether from a certain angle the problem looks like other cases that satisfy those conditions.

The question isn't whether or not we want to stick the fetus into an artificial category called "human", it's whether it has the specific features that make that category relevant to this particular problem in the first place.

See Leaky Generalizations and Replace The Symbol With The Substance

Comment author: Yvain 05 March 2011 04:46:11PM *  9 points [-]

Mechanized Deontic Logic is pretty okay, despite the dread I had because of the name. I'm no good at formal systems, but as far as I can understand it looks like a logic for proving some simple results about morality: the example they give is "If you should see to it that X, then you should see to it that you should see to it that X."

I can't immediately see a way this would destroy the human race, but that's only because it's nowhere near the point where it involves what humans actually think of as "morality" yet.

Comment author: Yvain 05 March 2011 05:03:18PM *  11 points [-]

Utilibot Project is about creating a personal care robot that will avoid accidentally killing its owner by representing the goal of "owner health" in a utilitarian way. It sounds like it might work for a robot with a very small list of potential actions (like "turn on stove" and "administer glucose") and a very specific list of owner health indicators (like "hunger" and "blood glucose level"), but it's not very relevant to the broader Friendly AI program.

Having read as many papers as I have time to before dinner, my provisional conclusion is that Vladimir Nesov hit the nail on the head

Comment author: Yvain 05 March 2011 04:34:21PM *  14 points [-]

Allen - Prolegomena to Any Future Moral Agent places a lot of emphasis on figuring out of a machine can be truly moral, in various metaphysical senses like "has the capacity to disobey the law, but doesn't" and "deliberates in a certain way". Not only is it possible that these are meaningless, but in a superintelligence the metaphysical implications should really take second-place to the not-getting-turned-into-paperclips implications.

He proposes a moral Turing Test, where we call a machine moral if it can answer moral questions indistinguishably from a human. But Clippy would also pass this test, if a consequence of passing was that the humans lowered their guard/let him out of the box. In fact, every unfriendly superintelligence with a basic knowledge of human culture and a motive would pass.

Utilitarianism considered difficult to implement because it's computationally impossible to predict all consequences. Given that any AI worth its salt would have a module for predicting the consequences of its actions anyway, and that the potential danger of the AI is directly related to how good this module is, that seems like a non-problem. It wouldn't be perfect, but it would do better than humans, at least.

Deontology, same problem as the last one. Virtue ethics seems problematic depending on the AI's motivation - if it were motivated to turn the universe to paperclips, would it be completely honest about it, kill humans quickly and painlessly and with a flowery apology, and declare itself to have exercised the virtues of honesty, compassion, and politeness? Evolution would give us something at best as moral as humans and probably worse - see the Sequence post about the tanks in cloudy weather.

Still not impressed.

Comment author: Yvain 05 March 2011 04:46:11PM *  9 points [-]

Mechanized Deontic Logic is pretty okay, despite the dread I had because of the name. I'm no good at formal systems, but as far as I can understand it looks like a logic for proving some simple results about morality: the example they give is "If you should see to it that X, then you should see to it that you should see to it that X."

I can't immediately see a way this would destroy the human race, but that's only because it's nowhere near the point where it involves what humans actually think of as "morality" yet.

Comment author: Yvain 05 March 2011 04:11:25PM *  23 points [-]

I started looking through some of the papers and so far I don't feel enlightened.

I've never been able to tell whether I don't understand Kantian ethics, or Kantian ethics is just stupid. Take Prospects For a Kantian Machine. The first part is about building a machine whose maxims satisfy the universalizability criterion: that they can be universalized without contradicting themselves.

But this seems to rely a lot on being very good at parsing categories in exactly the right way to come up with the answer you wanted originally.

For example, it seems reasonable to have maxims that only apply to certain portions of the population, for example: "I, who am a policeman, will lock up this bank robber awaiting trial in my county jail" generalizes to "Other policemen will also lock up bank robbers awaiting trial in their county jails" if you're a human moral philosopher who knows how these things are supposed to work.

But I don't see what's stopping a robot from coming up with "Everyone will lock up everyone else" or "All the world's policemen will descend upon this one bank robber and try to lock him up in their own county jails". After all, Kant universalizes "I will deceive this murderer so he can't find his victim" to "Everyone will deceive everyone else all the time" and not to "Everyone will deceive murderers when a life is at stake". So if a robot were to propose "I, a robot, will kill all humans", why should we expect it to universalize it to "Everyone will kill everyone else" rather than "Other robots will also kill all humans", which just means the robot gets help?

And even if it does universalize correctly, in the friendly AI context it need not be a contradiction! If this is a superintelligent AI we're talking about, then even in the best case scenario where everything goes right the maxim "I will try to kill all humans" will universalize to "Everyone will try to kill everyone else". Kant said this was contradictory in that every human will then be dead and none of them will gain the desserts of their murder - but in an AI context this isn't contradictory at all: the superintelligence will succeed at killing everyone else, the actions of the puny humans will be irrelevant, and the AI will be just fine.

(actually, just getting far enough to make either of those objections involves hand-waving away about thirty other intractable problems you would need just to get that far; but these seemed like the most pertinent).

I'll look through some of the other papers later, but so far I'm not seeing anything to make me think Eliezer's opinion of the state of the field was overly pessimistic.

Comment author: Yvain 05 March 2011 04:34:21PM *  14 points [-]

Allen - Prolegomena to Any Future Moral Agent places a lot of emphasis on figuring out of a machine can be truly moral, in various metaphysical senses like "has the capacity to disobey the law, but doesn't" and "deliberates in a certain way". Not only is it possible that these are meaningless, but in a superintelligence the metaphysical implications should really take second-place to the not-getting-turned-into-paperclips implications.

He proposes a moral Turing Test, where we call a machine moral if it can answer moral questions indistinguishably from a human. But Clippy would also pass this test, if a consequence of passing was that the humans lowered their guard/let him out of the box. In fact, every unfriendly superintelligence with a basic knowledge of human culture and a motive would pass.

Utilitarianism considered difficult to implement because it's computationally impossible to predict all consequences. Given that any AI worth its salt would have a module for predicting the consequences of its actions anyway, and that the potential danger of the AI is directly related to how good this module is, that seems like a non-problem. It wouldn't be perfect, but it would do better than humans, at least.

Deontology, same problem as the last one. Virtue ethics seems problematic depending on the AI's motivation - if it were motivated to turn the universe to paperclips, would it be completely honest about it, kill humans quickly and painlessly and with a flowery apology, and declare itself to have exercised the virtues of honesty, compassion, and politeness? Evolution would give us something at best as moral as humans and probably worse - see the Sequence post about the tanks in cloudy weather.

Still not impressed.

Comment author: Yvain 05 March 2011 04:11:25PM *  23 points [-]

I started looking through some of the papers and so far I don't feel enlightened.

I've never been able to tell whether I don't understand Kantian ethics, or Kantian ethics is just stupid. Take Prospects For a Kantian Machine. The first part is about building a machine whose maxims satisfy the universalizability criterion: that they can be universalized without contradicting themselves.

But this seems to rely a lot on being very good at parsing categories in exactly the right way to come up with the answer you wanted originally.

For example, it seems reasonable to have maxims that only apply to certain portions of the population, for example: "I, who am a policeman, will lock up this bank robber awaiting trial in my county jail" generalizes to "Other policemen will also lock up bank robbers awaiting trial in their county jails" if you're a human moral philosopher who knows how these things are supposed to work.

But I don't see what's stopping a robot from coming up with "Everyone will lock up everyone else" or "All the world's policemen will descend upon this one bank robber and try to lock him up in their own county jails". After all, Kant universalizes "I will deceive this murderer so he can't find his victim" to "Everyone will deceive everyone else all the time" and not to "Everyone will deceive murderers when a life is at stake". So if a robot were to propose "I, a robot, will kill all humans", why should we expect it to universalize it to "Everyone will kill everyone else" rather than "Other robots will also kill all humans", which just means the robot gets help?

And even if it does universalize correctly, in the friendly AI context it need not be a contradiction! If this is a superintelligent AI we're talking about, then even in the best case scenario where everything goes right the maxim "I will try to kill all humans" will universalize to "Everyone will try to kill everyone else". Kant said this was contradictory in that every human will then be dead and none of them will gain the desserts of their murder - but in an AI context this isn't contradictory at all: the superintelligence will succeed at killing everyone else, the actions of the puny humans will be irrelevant, and the AI will be just fine.

(actually, just getting far enough to make either of those objections involves hand-waving away about thirty other intractable problems you would need just to get that far; but these seemed like the most pertinent).

I'll look through some of the other papers later, but so far I'm not seeing anything to make me think Eliezer's opinion of the state of the field was overly pessimistic.

Comment author: Yvain 16 February 2011 12:17:26PM *  28 points [-]

You use the words "solved" and settled" here, but I think they have very different meanings. In particular I can think of two relevant definitions of "settled": first, that someone, somewhere, has the correct answer; but second, that the correct answer is widely accepted, uncontroversial, and someone ignorant of the field can easily discover it just by reading a textbook..

I think your examples fall into the first category but not the second. According to the PhilPapers survey, only 32% of philosophers "accept" physicalism (a further 20% were "leaning towards" it). Another presentation of the poll said that 73% of philosophers either "lean toward or accept atheism".

When you can't get even three quarters of a field to even "lean toward" a position, I don't think you can call that a "settled question" under the second definition, especially compared to science where hopefully 100% of astronomers would either "lean toward or accept" heliocentrism. And when I complain that philosophers cannot settle questions, I am mostly referring to that second definition.

In response to Secure Your Beliefs
Comment author: timtyler 12 February 2011 05:42:50PM *  18 points [-]

http://whatstheharm.net/ is interesting - but not very balanced.

I was reminded of http://www.quackwatch.com/ ... I looked at:

It is trivial to compile a list of harms done by conventional medicine as well. Doctors kill many people each year, What patients want is harms vs benefits. Lists of harms are anecdotal evidence - and alas are of very little use.

Comment author: Yvain 12 February 2011 10:28:57PM 11 points [-]

Yes; note also their page on Moon Landing Denial.

Comment author: Jack 06 February 2011 09:21:26PM *  3 points [-]

So the draw is unordered. I don't deserve any more karma than prase or Vaniver. That said, you've done a ton of work and I would have no problem putting lessons together for a post. On the other hand, you totally deserve more karma for the work you put in moderating this- so if you want it, take it. If I posted in the discussion section it would be pretty easy to make sure all players got the same karma. If I post on the main page that gets a lot harder.

I also think losing players should be awarded with karma if they can recount important lessons. People shouldn't have to give away insight without getting karma in return. So maybe we should just have a discussion thread where players talk about what they learned (and where the three of us can get victory karma)? And then you could just quote the worthwhile/top voted insights and put them in a post for the main page.

ETA: The other thing is the second game is still going. Since they might have their own lessons, the top level post could include those too- as they'll probably have their own discussion thread for their winner/winners.

Comment author: Yvain 06 February 2011 11:42:42PM 0 points [-]

My preference is to not be the one who has to arrange this, but if no one else wants to I will take the responsibility. If everyone else agrees on you, I'd support that.

Comment author: Yvain 06 February 2011 07:57:58PM 4 points [-]

A series of flashing lights seen from the moon have been discovered to be Morse code and translated into the following telegram:

Several interrogated eyewitnesses in Berlin report that the huge explosion that the invaders witnessed from a distance was, in fact, the flame of a so-called "rocket" taking the Kajser to safety along with a hand-picked staff. Long the personal project of Herr Prof. Doktor Weltschmerz (Dr.phil, Dr.med, D.Sc, D.Occ Psych, Eng.D, etc.), it was completed at the last moment and sparing no expenses. The last remnants of the German empire are now believed to have relocated to the dark side of the moon, where they will be licking their wounds and plotting revenge.

Unfortunately, as the loss of several mineral-rich German territories made better materials unavailable, the Kajser's men had to once again use up all of Berlin's accumulated truth reserves to be forged into plating for the rocket. This is a disappointment to the occupying troops, who had hoped for an opportunity to loot the city to replace the truth supplies that were expended while on the war path. It also helps explain why the news of the Kajser's escape were not heard earlier. New truth had to be imported from the coffers of the occupying nations and brought to the interrogation chambers to make the eyewitnesses talk of what actually happened. Before, only wild fabrications could be gotten out of them, such as mad ravings about a possible time when all the rulers weren't constantly seeking to stab each other in the back.

Comment author: Yvain 06 February 2011 07:55:09PM 0 points [-]

So, should we have a thread where we recount lessons and experiences? Should it be in Discussion or the main site? And should I post it, or should Jack?

Comment author: Wei_Dai 05 February 2011 05:04:57AM 6 points [-]

Perhaps individual philosophers honestly seek answers, but academic philosophy as an institution has evolved to not really seek answers. That is, mechanisms that promote the spread of true ideas have atrophied or were never built, while mechanisms to promote diversity of positions have proliferated, without anyone consciously wanting these things to happen.

(This is just a guess as I'm not really familiar with the inner workings of academia.)

Comment author: Yvain 06 February 2011 02:09:47PM 2 points [-]

Then what I'm wondering is what these mechanisms are, how they caught on, and how we can identify them in philosophy in other disciplines so we can be on guard.

Comment author: TheOtherDave 05 February 2011 06:01:21AM 2 points [-]

I think it's worth distinguishing between cases where being correct has pragmatic advantages, and cases where it doesn't.

For example, there are plenty of people who reject the philosophical underpinnings of modern medicine. There always have been. Those people don't go to medical school, don't go to conferences, etc.; eventually a whole discussion emerges to which they are not invited. Believers in medicine don't convince the believers in homeopathy, they ignore them and concentrate on doing medicine.

And because doctors achieve more valuable things more reliably than homeopaths, over time they displace the homeopaths... they create their own community within which a belief in medicine is pervasive. That the homeopaths are not convinced isn't actually important; it just means they aren't part of that community.

Of course, if believing in homeopathy doesn't correlate with skill at carpentry, then the medicine/homeopathy disagreement may continue to exist among carpenters. But so what? Who cares what carpenters think about medicine? Why is resolving that disagreement worth devoting energy to? Better for medicinists to devote their efforts to advancing medicine.

Similarly, there are plenty of people who reject atheism. There always will be. The thing for the atheists to do is work in areas where atheism gives them a pragmatic advantage. Over time they will displace the theists in that area, and the atheism/theism disagreement will disappear in that area. If atheism confers no demonstrable advantage to carpenters, then the disagreement will continue to exist among carpenters. But, again, so what?

By the same token, if a philosophical problem turns out to not have any pragmatic implications -- that is, if there is no area where people with the correct answer can do something valuable that people with the incorrect answer can't do, or can't do as well -- then the disagreement will continue to exist everywhere. But, again, so what?

Comment author: Yvain 06 February 2011 02:08:41PM 1 point [-]

Within the more practical sorts of philosophy, like logic, epistemology, and morality, there are potentially huge gains to society for getting it right. But these can only be "tested" (in the sense of creating a society that revolves around certain philosophical ideas) on a multi-decade time scale with a huge investment of resources and possible human suffering if you're wrong, and all experiments are necessarily imperfect (communists still argue their principles would have worked if the situation had been different).

That means there are practical gains from having good philosophy, but not in a way that means it can be decided by experiment.

Comment author: wedrifid 06 February 2011 05:23:58AM 5 points [-]

This seems more like Newcomb's to me in that whether or not your genes are passed on is linked to whether or not you ultimately decide to reproduce - so I think the Newcombness of the problem is legitimate.

I might suggest that whether or not your genes are passed on is linked to whether or not you decide to gain status and have a lot of sex. At the scale where we consider Azathoth birth control barely comes into consideration.

In fact, when playing this game with Azathoth it may best to go ahead and submit to your every promiscuous desire while at the same time being practical with contraception. The ability to follow your instincts while hypocritically gaming the system is a trait that Azathoth holds in high esteem and if we care to anthropomorphise him at all we must consider him as if he thinks as though it was still some time in the past we know he doesn't care much about condoms, pills and abortions. So it's like primary school all over - punch him in the face and then you can be friends.

Comment author: Yvain 06 February 2011 02:00:09PM 6 points [-]

If what you're saying is that genes directly affect promiscuity and sex drive, but are too low-resolution to directly affect things like whether or not you use condoms, that sounds like a good solution. Although it's not the least convenient possible world and I'm still curious whether an alien being with stronger genetic determinism should take this argument into account.

Comment author: TobyBartels 06 February 2011 03:21:57AM 1 point [-]

Whether my genes are passed on after me is linked to whether I reproduce, much as (in the relevant versions of Newcomb) the money in Box B is linked to whether I take Box A. But whether my genes were passed on before me is not linked to whether I reproduced, much as (in the smoking-lesion problem) my cancer status is not linked to whether I smoke.

Comment author: Yvain 06 February 2011 01:58:22PM 1 point [-]

Think of this at the genetic level, not the personal level. Let's say you have a gene G, which affects decision-making about reproduction. If G causes people to decide not to reproduce, then your ancestors possessing gene G will have not reproduced and you won't exist. If G makes you decide to reproduce, then your ancestors will have reproduced and you will exist. If we interpret decisions as altering the output of the algorithm that produced them, then deciding not to reproduce can alter the effects of gene G and therefore affect your ancestors with the gene.

Comment author: AlephNeil 06 February 2011 12:56:48AM *  4 points [-]

The 'Azathoth problem' is isomorphic to the smoking lesion problem, which is not isomorphic to Newcomb's problem.

Hence, any decision theory capable of both (i) one-boxing in Newcomb's problem and (ii) choosing to smoke in the 'smoking lesion' problem will have no difficulty here.

EDIT: I'd better sketch out this "isomorphism": "smoking" = "acting virtuously, in defiance of our evolutionary drives", "not smoking" = "giving in to our instincts and trying to optimize number of children". "having the lesion" = "carrying genes that predispose you to virtuous behaviour, and therefore having a smaller chance of having been born in the first place", "not having the lesion" = "carrying genes that predispose you to evolutionarily 'selfish' behaviour and therefore having a larger chance of having been born".

Comment author: Yvain 06 February 2011 03:08:50AM *  5 points [-]

This reminds me of a discussion a while back where I was interpreting Calvinist predestination as equivalent to Newcomb and Eliezer was interpreting it as equivalent to Smoking Lesion.

I think the difference involves whether the state is linked to a single urge or input, or whether it's linked to your entire decision-making process.

In the smoking lesion problem, your genotype is linked to whether or not you feel an urge to smoke. Once you feel the urge, the interesting decision theoretic bit is done; you can then decide whether or not to smoke knowing that it can't possibly affect your genotype.

In Newcomb's problem, the money in the boxes is linked to your final decision, so changing your decision can (in theory) change the money you find in the box.

This seems more like Newcomb's to me in that whether or not your genes are passed on is linked to whether or not you ultimately decide to reproduce - so I think the Newcombness of the problem is legitimate.

Philosophers and seeking answers

25 Yvain 04 February 2011 10:51PM

This thread has produced some interesting commentary around whether philosophers actually want to answer their own questions, or whether they enjoy sounding profound by debating big questions but don't want to lose that opportunity for profundity by finding single correct answers to them.

I don't quite disagree with the latter theory: the main reason I quit academic philosophy was exasperation that people were still debating questions where the right answer seemed obvious to me (like theism vs. atheism, or whether there was a universally compelling morality/aesthetics of pure reason), and worry that my philosophical career would involve continuing to debate these issues ad nauseum rather than helping to solve them and move on to the next problem.

But when I explained this to a particularly sarcastic friend, he summarized it as "So you think philosophy is useless because not everyone agrees with you?"

The problem isn't that philosophers never come up with solutions. The problem is that they come up with too many different solutions.

Science has solved many scientific problems, and anyone wondering what the solution is can look it up in a book or on Wikipedia. Philosophers have also solved many philosophical problems, but it is full of so many distractions and false solutions that anyone wondering which proposed solution is correct will have to become nearly as good a philosopher as the person who solved it in the first place. It's much easier for science to settle its disputes via experiment than for philosophy to settle its disputes via debate.

I am wary of criticizing the discipline of philosophy simply on the grounds that not everyone in it agrees with me. But I also don't want to let it off and say it's okay that they've managed to go so long without coming to any answers, when it seems to me that settling at least some of the easier problems is not that difficult.

How do we tell the difference between a discipline that doesn't really seek answers and a discipline which honestly seeks answers but just can't agree within itself? And how can philosophy do something about its level of internal disagreement without having to apply the "kick out everyone who disagrees with Less Wrong" solution?

Comment author: Yvain 04 February 2011 10:29:05PM *  23 points [-]

Proof by contradiction: suppose this method is true. Then anything which can be mocked is false. Consider the following mockery:

"Right, so political scientists and philosophers have debated these issues for thousands of years, and all of a sudden a humorist can figure out how to obtain a quick truth-value merely by using the average person's laughter as library, laboratory, and debate floor?

If this works, I envision elections run solely on attempts to mock the candidates involved. As Barack Obama makes a speech, a solemnly convened panel of Jay Leno, David Letterman, and Conan O'Brien will feverishly interrupt with the best one-liners they can craft. Recording devices will be in place all around the room to measure laughter volume to the decibel.

Then Obama's opponent gets up. As Letterman makes fun of his tax policies, the crowd goes wild. Fifty decibels, sixty...seventy decibels of laughter. As the chortling dies down, Chief Justice Roberts announces what everyone already knows - Obama will have four more years in power.

Gone are the economists and statisticians and bureaucrats, as forgotten as alchemists and astrologers. In there place rises the New De-mock-racy, in which comedy is recognized as the key to good policy. Laughter really is the best medicine - as is tangibly proven when doctors switch to making diagnoses solely by mocking how stupid all the other possibilities are. 'Dr. Johnson thinks you have Alzheimers'? He must have Alzheimers' himself to forget to account for your thyroid test results! HAHAHAHAHA!'

The world is transformed into utopia - until it's wiped out by a paper-clip-maximizing robot created after someone got the Singularity Institute closed by calling it 'rapture of the nerds' and got a good chuckle out of a few people who enjoyed sticking it to the eggheads."

Since the method can be mocked, it must be false. But then we have arrived at a contradiction: the method is both true and false. Therefore, the method is false.

Comment author: Jack 03 February 2011 02:19:57AM 0 points [-]

lyo retreats to spa

Note that the fleet in the Norwegian Sea did not participate in the attack on Norway because it was listed in the orders as A nwg instead of F nwg, and all players previously agreed I should be strict with mistakes.

Not that it matters but... when did this happen? You fixed mis-written orders in the past and I don't believe I ever agreed to anything.

In any case, the mis-communication was due to saboteurs loyal to King George.

Movement orders for the Fall 1911 turn are due 1800 GMT on Friday, 4/2/2011

Turkey and Austria... can we make this faster than that? I don't see why the game can't end tonight (or the next 12 hours if some of your are in a different hemisphere). England won't oppose the invasion of Berlin...

Comment author: Yvain 03 February 2011 02:28:58PM 0 points [-]

I'm thinking of a time Germany mixed up Burgundy and Gascony and I auto-corrected it. I made a post asking if I should continue doing so, and of the people who responded, all were opposed.

Comment author: wedrifid 01 February 2011 04:12:36AM *  2 points [-]

Why does observational knowledge work in your own possible worlds, but not in counterfactuals?

It does not work in this counterfactual. Omega could have specified the counterfactual such that the observational knowledge in the counterfactual was as usable as that in the 'real' world. (Most obviously by flat out saying it is so.)

The reason we cannot use the knowledge from this particular counterfactual is that we have no knowledge about how the counterfactual was selected. The 99% figure (as far as we know) is not at all relevant to how likely it is that we would be presented with an even or odd counterfactual result. When we intuitively reject the counterfactual result we, or at least I, are making this judgement.

Comment author: Yvain 01 February 2011 01:06:18PM *  0 points [-]

I agree with this answer. I believe in the question as given the answer is probably "even", but if Omega clarifies that in a counterfactual world randomly selected from the pool of all counterfactual worlds the calculator displayed "odd", then you should have a 50% probability each way.

The reason observational evidence works in your world but not in other not randomly selected possible worlds is that if Omega selected the world in any way other than at random, then we're talking about a world that may have been specifically selected for being improbable.

Comment author: Yvain 30 January 2011 11:33:43PM 0 points [-]

I don't have Turkey's build orders yet, so I am giving them a 24 hour grace period. If they don't have them in by the time I get home tomorrow, they waive the builds. Sorry for not being able to post the new turn tonight.

Comment author: datadataeverywhere 30 January 2011 04:56:51AM *  2 points [-]

I do not think I am quite addressing your question. Specifically, I don't think there has been a wide enough discussion about calibration for there to be a single widely accepted method.

However, what I would like to point out is that a single-number calibration necessarily discards information, and there is no one true way to decide which information to discard.

A gets binary questions right 98% of the time, but expects to get them correct 99% of the time. B gets binary questions right 51% of time time, but expects to get them correct 52% of the time.

In some cases, A and B must be treated as equally calibrated (Zut Allais! is relevant). In some cases, B can be considered much better calibrated, and in almost all cases we don't care either way, because B's predictions are almost never useful, whereas A's almost always are.

Even this is a dramatic simplification, painfully restricting our information about the situation. Perhaps A never has false positives; or maybe B never has false positives! This is extremely relevant to many questions, but can't be represented in any single-number metric.

No matter what your purpose, domain knowledge matters, and I suspect that calibration does not carry over well from one domain to another, so finding out that you know little history but are well calibrated to how poorly you know things will not help you evaluate how reliable your predictions in your primary field are.

Binary questions are usually already horribly under-sampled. We can ask binary questions about history, but it probably matters in the real world whether your answer was 2172 or 1879 if the correct answer was 1880. Ideally, we could provide a probability distribution for the entire range of incorrectness, but in practice, I think the best measure is to report the false positive and false negative rate of an agent on a set of questions along with their own estimates for their performance on those questions. I realize this is four times as many numbers as you want, but you can then condense them however you like, and I really think that the 4-tuple is more than four times more useful than any single-number measure!

Do you have a more specific purpose in mind? I'm curious what spurred your question.

Comment author: Yvain 30 January 2011 01:37:52PM 1 point [-]

Do you have a more specific purpose in mind? I'm curious what spurred your question.

A prof doing an experiment gave me a bunch of data from calibration tests with demographic identifiers, and I'd like to be able to analyze it to say things like "Old people have better calibration than young people" or "Training in finance improves your calibration".

Looking for information on scoring calibration

8 Yvain 29 January 2011 10:24PM

There are lots of scoring rules for probability assessments. Log scoring is popular here, and squared error also works.

But if I understand these correctly, they are combined measurements of both domain-ability and calibration. For example, if several people took a test on which they had to estimate their confidence in their answers to certain true or false questions about history, then well-calibrated people would have a low squared error, but so would people who know a lot about history.

So (I think) someone who always said 70% confidence and got 70% of the questions right would get a higher score than someone who always said 60% confidence and got 60% of the questions right, even though they are both equally well calibrated.

The only pure calibration estimates I've ever seen are calibration curves in the form of a set of ordered pairs, or those limited to a specific point on the cuve (eg "if ey says ey's 90% sure, ey's only right 60% of the time"). There should be a way to take the area under (or over) the curve to get a single value representing total calibration, but I'm not familiar with the method or whether it's been done before. Is there an accepted way to get single-number calibration scores separate from domain knowledge?

Comment author: Jack 26 January 2011 06:00:17PM 1 point [-]

the new, technologically advanced British ships

Please tell me I get some kind of force multiplier.

Comment author: Yvain 27 January 2011 11:03:13PM 3 points [-]

You get a force multiplier of one.

Comment author: prase 21 January 2011 01:21:35AM 1 point [-]

Austria protests! The order was Vie-Bud, not Vie-Boh.

Comment author: Yvain 21 January 2011 04:36:59PM 0 points [-]

I'll check and if you're right I'll fix it over winter.

Comment author: Kaj_Sotala 21 January 2011 07:50:44AM 1 point [-]

I didn't get e-mail notification for this turn or the previous one. Did you stop sending them?

Comment author: Yvain 21 January 2011 04:36:46PM 0 points [-]

No, I'm sending them as normal. Has anyone else had this problem?

Comment author: Vaniver 19 January 2011 08:14:48PM 0 points [-]

Can we bump the deadline back to 1800 GMT? I get home at about 1650 GMT and would like to be able to go over things and send in my orders after that, rather than the night before.

(i.e. I plan to send them in late like last time but would feel better with a sanction :P )

Comment author: Yvain 20 January 2011 02:51:57PM 0 points [-]

Go ahead.

Comment author: Yvain 14 January 2011 04:30:45PM 0 points [-]

I am still waiting for Austria's orders.

Comment author: mkehrt 08 January 2011 06:41:58PM 2 points [-]

You said pretty much exacty everything I would have said and more.

One question--I only read the first third of so and skimmed the rest. The bits I read seemed to give a false dichotomy for dates of the composition of the gospels. The authors discussed atheistic schools that believed the gospels were all composed post 100 and contrasted these with the pre70 dates of Christian belief. Do they ever discuss the modern scholarly mostly-consensus of 70-90?

Relatedly, do you know of any good arguments for post 70 composition dates, especially for Matthew and Luke, other than fulfilled prophecies of the destruction of Jerusalem? I've always found the arguments that these books were written before 70 because they could not have predicted the destruction of Jerusalem suspiciously question-begging about the possibility of miracles.

Comment author: Yvain 09 January 2011 07:45:39PM 1 point [-]

I'm afraid I really know very little about dating the Gospels; I just trusted what I saw on Wikipedia.

Comment author: Yvain 08 January 2011 06:02:14PM *  58 points [-]

I read...a surprisingly large amount of that.

If I understand it right, they are saying that modern scholarship confirms that the Gospels avoid certain obvious failure modes - eg being written hundreds of years after the fact, wildly contradicting each other on important points, and erring on simple points of geography and history - and that someone would've called them on it if they just blatantly made things up - therefore the Gospels can be assumed mostly true. The Gospels say many people saw Jesus die on the Cross and then saw him alive later, and that natural explanations (Jesus survived the crucifixion, everyone was hallucinating, it was Jesus' twin brother - yes, they actually addressed that) are all unconvincing; therefore Jesus really was resurrected. According to the Gospels, this was seen by many witnesses, including luminaries like St. Peter, and none of them later came forward to say "No, we didn't see this at all, shut up". Further, many of them later died an extremely predictable martyr's death, proving that they believed in Christ's resurrection enough to sacrifice their lives for him, something they wouldn't have done if it were all made up (they point out that although some people, like kamikaze pilots, have sacrificed their lives to false philosophies, it is far more unlikely that the Apostles would sacrifice their lives to a false empirical fact, namely that they had seen Jesus rise from the dead).

Multiplying the low probabilities of everyone involved simultaneously having some kind of fit of insanity leading them to sincerely believe Jesus had risen from the dead gives 1 : 10^39 against, and since this is a very small number obviously the argument must be correct.

This argument doesn't quite take the truth of the Gospels as a premise, but it comes close. Although there are some atheist accounts that allow for the truth of the Gospels as written while still casting doubt on Christ's divinity, that's not where the smart money lies - most atheists would deny to one degree or another the validity of the Gospels themselves. Either the entire thing was made up (a theory which the McGrews reject, and I think rightly) or a historical Jesus had various miracles falsely attributed to him by overzealous believers. This leaves the McGrews' objection that the existence of a wider Christian community, many of whom had been personally involved in the events described, would have limited the Gospel writers' ability to make things up even if they had been so inclined.

So instead of basing his argument on the likelihood of people hallucinating resurrected Jesus, the McGrews should have investigated the probability that the Gospel writers would make up miracles and the probability that they would be caught; something like

P(resurrection) ~= P(gospels true) ~= 1 - [P(people make stuff up about Jesus) * P(they don't get called on it)]

So what is the probability that, given some historical tradition of Jesus, it will get embellished with made-up miracles and people will write gospels about it? Approximately 1: both Christians and atheists agree that the vast majority of the few dozen extant Gospels are false, including the infancy gospels, the Gospel of Judas, the Gospel of Peter, et cetera. All of these tend to take the earlier Gospels and stories and then add a bunch of implausible miracles to them. So we know that the temptation to write false Gospels laden with miracles was there. Apologists say that the four canonical Gospels are earlier and more official than the apocryphal Gospels, and I agree, but given the existence of a known tendency for people to make up books, and a set of books that sound made-up, the difference seems more one of degree than of kind.

That leaves the question of whether anyone would notice. The dates of all the Gospels are uncertain, but around 70 - 80 AD for the synoptics seems like a fair guess. The average life expectancy in classical Judaea for those who survived childhood was 40 to 50. That means Jesus' generation would be long gone by the time the first Gospel came out, and even people who were teenagers at the time of Jesus' crucifixion would be dying off. Christian tradition lists all the Apostles except John as dead by 75 AD.

There's also the more general question of argument from silence. Let's say someone did have evidence against something in the Gospels. Most Judeans at the time wouldn't have been literate, especially not in the Greek in which the Gospels were written. Many who were, might not have had the time or interest to pen responses to what seemed a minor cult at the time. If any did, those responses might not have spread in an age when every work had to be laboriously copied by hand. And if by some miracle a refutation did become popular, there's no reason to think we would know about it since many of the popular works of the age have been lost completely.

Matthew mentions that on the day of Jesus' crucifixion, graves opened and the dead walked the earth throughout the city of Jerusalem for several hours. No one else (including the other evangelists!) mentioned the dead walking the earth, either to confirm or refute it, so clearly the 1st century AD Judean skeptical community wasn't exactly on top of its game. That alone casts suspicion on the whole "if this was false, someone would've said so" argument.

All of this makes the Gospel argument relatively uninteresting to me. But it hints at a different problem which is interesting. Twenty years after the death of Christ, we have Paul writing letters to flourishing churches all across the eastern Mediterranean, all of whom seem to have at least a vague tradition of Christ being resurrected and appearing to people. That means Christianity spread really, really fast, presumably by people who were pretty sure they had met the resurrected Christ. At my current, limited level of Biblical scholarship I consider myself still confused on this point and yet to see a satisfactory explanation (people rising from the dead doesn't count as 'satisfactory').

Comment author: Kaj_Sotala 07 January 2011 09:26:05AM *  1 point [-]

Silesia retreats to Prussia.

Also, the set of orders I submitted (just rechecked) had the fleet in Sweden moving to Bal, that doesn't seem to have happened?

Comment author: Yvain 07 January 2011 05:25:31PM 0 points [-]

You're right. I've updated the second map to take account of both these points.

Comment author: orthonormal 05 January 2011 06:11:22PM 0 points [-]

I think your link to "Near Mode" is supposed to be a link to "Far Mode", and that you meant to link the following phrase to "Near Mode".

Comment author: Yvain 06 January 2011 06:26:17AM 0 points [-]

The link redirected to "near/far thinking", but I've changed it to reflect that.

Techniques for probability estimates

58 Post author: Yvain 04 January 2011 11:38PM

Utility maximization often requires determining a probability of a particular statement being true. But humans are not utility maximizers and often refuse to give precise numerical probabilities. Nevertheless, their actions reflect a "hidden" probability. For example, even someone who refused to give a precise probability for Barack Obama's re-election would probably jump at the chance to take a bet in which ey lost $5 if Obama wasn't re-elected but won $5 million if he was; such decisions demand that the decider covertly be working off of at least a vague probability.

When untrained people try to translate vague feelings like "It seems Obama will probably be re-elected" into a precise numerical probability, they commonly fall into certain traps and pitfalls that make their probability estimates inaccurate. Calling a probability estimate "inaccurate" causes philosophical problems, but these problems can be resolved by remembering that probability is "subjectively objective" - that although a mind "hosts" a probability estimate, that mind does not arbitrarily determine the estimate, but rather calculates it according to mathematical laws from available evidence. These calculations require too much computational power to use outside the simplest hypothetical examples, but they provide a standard by which to judge real probability estimates. They also suggest tests by which one can judge probabilities as well-calibrated or poorly-calibrated: for example, a person who constantly assigns 90% confidence to eir guesses but only guesses the right answer half the time is poorly calibrated. So calling a probability estimate "accurate" or "inaccurate" has a real philosophical grounding.

There exist several techniques that help people translate vague feelings of probability into more accurate numerical estimates. Most of them translate probabilities from forms without immediate consequences (which the brain supposedly processes for signaling purposes) to forms with immediate consequences (which the brain supposedly processes while focusing on those consequences).



Prepare for Revelation

What would you expect if you believed the answer to your question were about to be revealed to you?

In Belief in Belief, a man acts as if there is a dragon in his garage, but every time his neighbor comes up with an idea to test it, he has a reason why the test wouldn't work. If he imagined Omega (the superintelligence who is always right) offered to reveal the answer to him, he might realize he was expecting Omega to reveal the answer "No, there's no dragon". At the very least, he might realize he was worried that Omega would reveal this, and so re-think exactly how certain he was about the dragon issue.

This is a simple technique and has relatively few pitfalls.


Bet on it

At what odds would you be willing to bet on a proposition?

Suppose someone offers you a bet at even odds that Obama will be re-elected. Would you take it? What about two-to-one odds? Ten-to-one? In theory, the knowledge that money is at stake should make you consider the problem in "near mode" and maximize your chances of winning.

The problem with this method is that it only works when utility is linear with respect to money and you're not risk-averse. In the simplest case I should be indifferent to a $100,000 bet at 50% odds that a fair coin would come up tails, but in fact I would refuse it; winning $100,000 would be moderately good, but losing $100,000 would put me deeply in debt and completely screw up my life. When these sorts of consideration become paramount, imagining wagers will tend to give inaccurate results.


Convert to a Frequency

How many situations would it take before you expected an event to occur?

Suppose you need to give a probability that the sun will rise tomorrow. "999,999 in a million" doesn't immediately sound wrong; the sun seems likely to rise, and a million is a very high number. But if tomorrow is an average day, then your probability will be linked to the number of days it will take before you expect that the sun will fail to rise on at least one. A million days is three thousand years; the Earth has existed for far more than three thousand years without the sun failing to rise. Therefore, 999,999 in a million is too low a probability for this occurrence. If you think the sort of astronomical event that might prevent the sun from rising happens only once every three billion years, then you might consider a probability more like 999,999,999,999 in a trillion.

In addition to converting to a frequency across time, you can also convert to a frequency across places or people. What's the probability that you will be murdered tomorrow? The best guess would be to check the murder rate for your area. What's the probability there will be a major fire in your city this year? Check how many cities per year have major fires.

This method fails if your case is not typical: for example, if your city is on the losing side of a war against an enemy known to use fire-bombing, the probability of a fire there has nothing to do with the average probability across cities. And if you think the reason the sun might not rise is a supervillain building a high-tech sun-destroying machine, then consistent sunrises over the past three thousand years of low technology will provide little consolation.

A special case of the above failure is converting to frequency across time when considering an event that is known to take place at a certain distance from the present. For example, if today is April 10th, then the probability that we hold a Christmas celebration tomorrow is much lower than the 1/365 you get by checking on what percentage of days we celebrate Christmas. In the same way, although we know that the sun will fail to rise in a few billion years when it burns out its nuclear fuel, this shouldn't affect its chance of rising tomorrow.


Find a Reference Class

How often have similar statements been true?

What is the probability that the latest crisis in Korea escalates to a full-blown war? If there have been twenty crisis-level standoffs in the Korean peninsula in the past 60 years, and only one of them has resulted in a major war, then (war|crisis) = .05, so long as this crisis is equivalent to the twenty crises you're using as your reference class.

But finding the reference class is itself a hard problem. What is the probability Bigfoot exists? If one makes a reference class by saying that the yeti doesn't exist, the Loch Ness monster doesn't exist, and so on, then the Bigfoot partisan might accuse you of assuming the conclusion - after all, the likelihood of these creatures existing is probably similar to and correlated with Bigfoot. The partisan might suggest asking how many creatures previously believed not to exist later turned out to exist - a list which includes real animals like the orangutan and platypus - but then one will have to debate whether to include creatures like dragons, orcs, and Pokemon on the list.

This works best when the reference class is more obvious, as in the Korea example.


Make Multiple Statements

How many statements could you make of about the same uncertainty as a given statement without being wrong once?

Suppose you believe France is larger than Italy. With what confidence should you believe it? If you made ten similar statements (Germany is larger than Austria, Britain is larger than Ireland, Spain is larger than Portugal, et cetera) how many times do you think you would be wrong? A hundred similar statements? If you think you'd be wrong only one time out of a hundred, you can give the statement 99% confidence.

This is the most controversial probability assessment technique; it tends to give lower levels of confidence than the others; for example, Eliezer wants to say there's a less than one in a million chance the LHC would destroy the world, but doubts he could make a million similar statements and only be wrong once. Komponisto thinks this is a failure of imagination: we imagine ourselves gradually growing tired and making mistakes, whereas this method only works if the accuracy of the millionth statement is exactly the same as the first.

In any case, the technique is only as good as the ability to judge which statements are equally difficult to a given statement. If I start saying things like "Russia is larger than Vatican City! Canada is larger than a speck of dust!" then I may get all the statements right, but it won't mean much for my Italy-France example - and if I get bogged down in difficult questions like "Burundi is larger than Equatorial Guinea" then I might end up underconfident. In cases where there is an obvious comparison ("Bob didn't cheat on his test", "Sue didn't cheat on her test", "Alice didn't cheat on her test") this problem disappears somewhat.


Imagine Hypothetical Evidence

How would your probabilities adjust given new evidence?

Suppose one day all the religious people and all the atheists get tired of arguing and decide to settle the matter by experiment once and for all. The plan is to roll an n-sided numbered die and have the faithful of all religions pray for the die to land on "1". The experiment will be done once, with great pomp and ceremony, and never repeated, lest the losers try for a better result. All the resources of the world's skeptics and security forces will be deployed to prevent any tampering with the die, and we assume their success is guaranteed.

If the experimenters used a twenty-sided die, and the die comes up 1, would this convince you that God probably did it, or would you dismiss the result as a coincidence? What about a hundred-sided die? Million-sided? If a successful result on a hundred-sided die wouldn't convince you, your probability of God's existence must be less than one in a hundred; if a million-sided die would convince you, it must be more than one in a million.

This technique has also been denounced as inaccurate, on the grounds that our coincidence detectors are overactive and therefore in no state to be calibrating anything else. It would feel very hard to dismiss a successful result on a thousand-sided die, no matter how low the probability of God is. It might also be difficult to visualize a hypothetical where the experiment can't possibly be rigged, and it may be unfair to force subjects to imagine a hypothetical that would practically never happen (like the million-sided die landing on one in a world where God doesn't exist).



These techniques should be experimentally testable; any disagreement over which do or do not work (at least for a specific individual) can be resolved by going through a list of difficult questions, declaring confidence levels, and scoring the results with log odds. Steven's blog has some good sets of test questions (which I deliberately do not link here so as to not contaminate a possible pool of test subjects); if many people are interested in participating and there's a general consensus that an experiment would be useful, we can try to design one.

Comment author: Yvain 02 January 2011 07:54:44PM 0 points [-]

I am missing Germany and England's orders :(

Comment author: wedrifid 28 December 2010 08:53:57AM *  1 point [-]

To some degree, this article is less about moralizing and more of a "how to" guide.

The specific quote the grandparent was replying to is about moralizing.

And likewise, there is only one best charity: the one that helps the most people the greatest amount per dollar.

One could strip the moralizing element from the quote (and the article) in a fairly straightforward manner. The best charity someone can donate to is subjectively objective: the one that achieves the most benefit per dollar according to that persons values, altruistic or otherwise.

Comment author: Yvain 28 December 2010 09:14:03AM *  2 points [-]

The specific quote the grandparent was replying to is about moralizing.

The problem with the word "best" there is the same problem the word "good" always runs into - the difference between "a good car" and "a good person". I'm using "best charity" in the same sense I would use "best Arctic survival gear" - best at achieving the purpose you are assumed to have. Although I think there is a case for that also being the morally best for most moral systems in which "morally best" makes sense, that would be way outside the scope of this discussion.

Comment author: PhilGoetz 27 December 2010 08:28:39PM *  2 points [-]

And likewise, there is only one best charity: the one that helps the most people the greatest amount per dollar.

I disagree. Giving money to charity is not different from spending money on a latte at Starbucks. I spend money according to my values. And I still buy lattes. I am not Zachary Baumkletterer. Even Jesus said, "The poor you will have with you always", to justify spending an INCREDIBLE amount of money (enough to buy ten people's entire lives, in an era with no inflation, making it comparable to ten million US dollars today) on pouring perfume once on Jesus' feet. The guy was tired and depressed and about to be crucified and wanted his damn perfume, like I want my damn latte.

Similarly, people who gave money to keep a painting in a museum, might also spend considerably more money to buy paintings to hang in their houses, than it would take to save a life in another country. These people value art, and they value benefitting others. Draw a 2D plot, and label the axes "selfish ... unselfish" and "spiritual ... physical" ("spiritual" standing for art and other "impractical" values). One person might

  • buy a painting to hang in their bedroom (spiritual, selfish)
  • buy a painting to hang in their guest room (spiritual, sorta selfish)
  • spend to preserve a painting in a museum (spiritual, unselfish)
  • buy fuzzy slippers (physical, selfish)
  • spend money for vaccines in Africa (physical, unselfish)

And each of those things could have similar utility for them.

I don't think this is irrational. Irrational is spending any money at all on "charity" instead of spending it according to your utility function.

This post contains the hidden presupposition that charity, using a collective utility function, is more moral than self-oriented actions; and therefore, following our utility functions is immoral. This is an assertion about morality and rationality that has huge implications! It is resonant with a very common meme that says that "moral" behavior is behavior that we don't want to do, because we are fundamentally immoral. I say, instead, that morals are part of our utility function - that we have these things called morals because part of us really wants to be nice to other people. They are just another part of our utility function.

Encouraging unselfish behavior can be done by manipulating peoples' selfish desires to produce "unselfish" behavior (give to charity and get social benefits, or stay out of Hell), as a mechanism to solve PD problems with a given payoff matrix. But it can also be done by treating people in ways that encourage what natural unselfish tendencies they have - solving PD problems by changing people's payoff matrices.

Apply Kant's imperative. This post suggests that we have 2 utility functions, one for everyday life, and another for charity; and that the one for charity is more moral. But if everyone used such a charity utility function for everything they did, it would result in a global race to the bottom as economies imploded after spending all national wealth on ameliorating suffering while undercutting all private motivation. Therefore, it is less moral. It is not only not obviously moral, it is immoral, if that means anything, for a government, or a person, to spend every last dollar on helping the unfortunate before spending any money on education, roads, defense, art, or even entertainment.

Comment author: Yvain 28 December 2010 08:00:31AM *  14 points [-]

To some degree, this article is less about moralizing and more of a "how to" guide. If you want to help people, this is how to do it. If you don't want to help people, and you prefer to have lattes or works of fine art or whatever, then a how-to guide on how to help people isn't relevant to your interests.

To the degree that it is more than that, the article is an attempt to expose certain thought processes into consciousness so that they can be evaluated by conscious systems. People may be donating to these inefficient charities because they feel like it and they don't examine their feelings, even though if they were to consciously think the problem through they would give to more efficient charities. If, after realizing that the choice is between one kid's life or 1/1000 of a painting, someone still prefers the painting, I don't really have anything more I can say - but my guess is that's not a lot of the population.

You made a really good point in your mysticism post on Discussion, about the difference between categorizing things by their causes and categorizing things by their effects. When you talk about spiritual and unselfish choices, you're categorizing things by their causes - a donation to the painting come from the same warm feelings that also produce a donation to vaccines.

Efficient charity is about categorizing things by their effects - it doesn't matter how noble the feelings that produced a certain action, only how much that action did what you wanted it to do. If you want to help people, it's about how many people you helped.

Categorizing things by their causes is an academic activity that can only declare some people to be more "unselfish" than others and accord them bragging rights. In my opinion this doesn't have as much to do with the actual work of saving the world as categorizing things by effects. You say this article claims things about morality, but that's really not its purpose. Its purpose is - if you've seen all sorts of horrible things in the world, and it's reached the point where you're so mad you don't care what can or can't be classified as moral, you just want to fix things as quickly as possible, what do you do then?

I think the idea of something to protect is relevant here.

Comment author: Vaniver 28 December 2010 07:43:52AM *  1 point [-]

Hm, I was going to let Kropotkin live (but still put the rest of the royal family to death), letting him try out his ideas in a Siberian commune. I guess I should have ordered that beforehand. (Also, it isn't called Istanbul yet.)

Comment author: Yvain 28 December 2010 07:48:20AM *  0 points [-]

Fixed. Feel free to include role-playing points like that in your orders in the future if you want.

Comment author: Yvain 27 December 2010 10:56:57PM 18 points [-]

New Year's resolution is not to donate to things until I check if there's a matching donation drive starting the next week :( Anyway, donated a little extra because of all the great social pressure from everyone's amazing donations here. Will donate more when I have an income.

Comment author: Yvain 27 December 2010 09:45:57AM 4 points [-]

I cleared my history/cache in Firefox and the problem went away. Don't know why.

Comment author: Elizabeth 25 December 2010 05:15:39AM 22 points [-]

I find I run into a conundrum on this question, because there is a bias I fear overcompensating for. I know as a human that I am biased to care more about the one person standing in front of me than those ten thousand people starving in India that I'll never meet, but I find it difficult to apply that information. I know that donating money to, say, those malaria nets, will probably save more lives than donating to, say, my local food pantry. By these arguments, it seems that that fact should trump all, and I should donate to those malaria nets.

However, I know that my local food pantry is an organization that feeds people who really need food, that it has virtually no overhead, and that there are children who would be malnourished without it. I also know that there are people all over the world who will contribute to malaria nets, but it is highly unlikely that anyone outside my community will contribute to my local food pantry.

I agree that it is vitally important to think carefully about how we spend our charity money, and I understand that the difficulty I am having with this topic is an indication that I need to think more deeply on it, but I keep coming up against two basic issues.

  • There is no simple metric for "most good done." What if one disease costs little to prevent death, but leaves survivors crippled, while another costs much more to prevent death but leaves people healthy? Should I donate to the first, and burden the communities with many cripples, or to the second, and let people die? With food and medical care costing more in the developed world, should I only donate to help those in the undeveloped world, where my dollar will go farther?

  • Should I feel guilty for donating money to public radio because it doesn't save children? No. My purpose in donating money to public radio is to keep my favorite shows on the air, and my donations do that very efficiently. Yes, the money could go to save children, but so could the money I use to pay my cable bill. I should perhaps not consider it as charity the way I do a donation that saves children, but I should not feel guilty. If I have $500 allocated for entertainment and $500 allocated for charity, perhaps it should come out of the former. However, it would be disingenuous to say that donations for more frivolous causes, such as saving artwork, could be donated to better causes, such as malaria nets, unless we also point out that what we spent on our fancy dinner or our new dress or going to the movies could also be thus allocated.

Comment author: Yvain 25 December 2010 08:32:52PM *  6 points [-]

The first question is hard but not confusing (I'd say "yes" to the developing world example, though); the second question confuses me too and I don't have a good answer.

I think this whole "efficient charity" field is working in the tradition of utility theory, where people's desires are treated as givens and the only interesting question is how to maximize achievement of those desires.

In that context, if you desire getting nice clothes with strength X, and desire helping other people with strength Y, then you divide your resources accordingly and try to maximize the niceness of the clothes you get with X resources and the number of people you help with Y resources. In that model, "try and help as many people as you can per charity dollar" is about all you can say.

This is a terribly oversimplified model, both because desires might be more complicated (your desire might not be to help people, but to help Americans, or to help people who enjoy public radio like you do), and because people are not utilitarian agents and it is possible to change the strength of your desires. A model that takes those into account would have to, among other things, fully understand morality and what it means to "want" something, and I don't fully understand either, though they're both research interests.

So this essay is only about how to avoid one particularly obvious mistake that's easy to model in utility theory, and not about how to avoid more important moral and psychological mistakes.

On the harder problems, without having much philosophical foundation for doing so, I recommend Giving What We Can

Comment author: Psy-Kosh 25 December 2010 01:59:17PM 7 points [-]

You know... that actually seems like potentially a good idea. Not just a tongue in cheek style good idea, but I'm thinking that this could be an actually for real good idea, and not just as a way to make "those other people" see what they're doing. I'd want this implemented as a way to make it easier for me to keep such things in mind!

(The "infected wounds" link is broken, though, so mind explaining the concept re that?)

The only real difficulty that I see is that as things change (tech, economic conditions, etc), the actual cost of saving a child and the relative costs of saving a child vs saving puppies, etc might shift around. So you'd need some way to dynamically rename chunks of the currency. For instance, if improving tech and such leads to the equivalent of 400$ being sufficient to save a child, then what was called a DC would have to be renamed 2 DC.

This would be confusing.

Comment author: Yvain 25 December 2010 08:19:56PM *  4 points [-]

The "infected wound" originally linked to some organization that donated first aid kits to those who couldn't afford them. I'll try to fix that next time I update the site.

Comment author: jsteinhardt 25 December 2010 04:08:04AM 1 point [-]

This is an excellent article. A quick comment on one of the sentences:

GiveWell.org, a site which collects and interprets data on the effectiveness of charities, predicts that antimalarial drugs save one child from malaria per $5,000 worth of bed medicine, but insecticide-treated bed nets save one child from malaria per $500 worth of drugs.

I find this sentence somewhat confusing. Should the "worth of bed medicine" be just "medicine", and the "worth of drugs" be "worth of netting"?

Comment author: Yvain 25 December 2010 05:13:21AM 2 points [-]

Thank you, I have no idea what happened to my brain there.

Comment author: James_Miller 25 December 2010 01:12:37AM 40 points [-]

I'm going to assign this to my introductory microeconomics students to help them understand opportunity costs.

Comment author: Yvain 25 December 2010 03:12:21AM *  41 points [-]

That sort of terrifies me, but in a good way.

At the risk of tooting my own horn, this essay only incidentally addresses opportunity costs, but I wrote another essay a few years ago in a different style that addresses them more directly: A Modest Proposal

Comment author: Psy-Kosh 25 December 2010 01:31:47AM 2 points [-]

Very well presented.

Just a minor technical note: All the links that linked to other LW pages are broken. It looks like somehow the links ended up having those articles's names being appended to the link for this one.

For instance, the one that was supposed to link to money being the unit of caring instead tries to link to this: http://lesswrong.com/lw/3gj/efficient_charity_do_unto_others/lw/65/money_the_unit_of_caring/

Comment author: Yvain 25 December 2010 03:12:03AM 1 point [-]

Hm, never seen that particular error before. Thanks and fixed.

Comment author: Sniffnoy 24 December 2010 09:34:40PM 0 points [-]

Edit, find the "post to" box.

Comment author: Yvain 24 December 2010 10:12:16PM 0 points [-]

I see this box on some articles I post to Discussion, but not others.

Efficient Charity: Do Unto Others...

130 Post author: Yvain 24 December 2010 09:26PM

This was originally posted as part of the efficient charity contest back in November. Thanks to Roko, multifoliaterose, Louie, jmmcd, jsalvatier, and others I forget for help, corrections, encouragement, and bothering me until I finally remembered to post this here.

Imagine you are setting out on a dangerous expedition through the Arctic on a limited budget. The grizzled old prospector at the general store shakes his head sadly: you can't afford everything you need; you'll just have to purchase the bare essentials and hope you get lucky. But what is essential? Should you buy the warmest parka, if it means you can't afford a sleeping bag? Should you bring an extra week's food, just in case, even if it means going without a rifle? Or can you buy the rifle, leave the food, and hunt for your dinner?

And how about the field guide to Arctic flowers? You like flowers, and you'd hate to feel like you're failing to appreciate the harsh yet delicate environment around you. And a digital camera, of course - if you make it back alive, you'll have to put the Arctic expedition pics up on Facebook. And a hand-crafted scarf with authentic Inuit tribal patterns woven from organic fibres! Wicked!

...but of course buying any of those items would be insane. The problem is what economists call opportunity costs: buying one thing costs money that could be used to buy others. A hand-crafted designer scarf might have some value in the Arctic, but it would cost so much it would prevent you from buying much more important things. And when your life is on the line, things like impressing your friends and buying organic pale in comparison. You have one goal - staying alive - and your only problem is how to distribute your resources to keep your chances as high as possible. These sorts of economics concepts are natural enough when faced with a journey through the freezing tundra.


But they are decidedly not natural when facing a decision about charitable giving. Most donors say they want to "help people". If that's true, they should try to distribute their resources to help people as much as possible. Most people don't. In the "Buy A Brushstroke" campaign, eleven thousand British donors gave a total of £550,000 to keep the famous painting "Blue Rigi" in a UK museum. If they had given that £550,000 to buy better sanitation systems in African villages instead, the latest statistics suggest it would have saved the lives of about one thousand two hundred people from disease. Each individual $50 donation could have given a year of normal life back to a Third Worlder afflicted with a disabling condition like blindness or limb deformity..

Most of those 11,000 donors genuinely wanted to help people by preserving access to the original canvas of a beautiful painting. And most of those 11,000 donors, if you asked, would say that a thousand people's lives are more important than a beautiful painting, original or no. But these people didn't have the proper mental habits to realize that was the choice before them, and so a beautiful painting remains in a British museum and somewhere in the Third World a thousand people are dead.

If you are to "love your neighbor as yourself", then you should be as careful in maximizing the benefit to others when donating to charity as you would be in maximizing the benefit to yourself when choosing purchases for a polar trek. And if you wouldn't buy a pretty picture to hang on your sled in preference to a parka, you should consider not helping save a famous painting in preference to helping save a thousand lives.

Not all charitable choices are as simple as that one, but many charitable choices do have right answers. GiveWell.org, a site which collects and interprets data on the effectiveness of charities, predicts that antimalarial drugs save one child from malaria per $5,000 worth of medicine, but insecticide-treated bed nets save one child from malaria per $500 worth of netting. If you want to save children, donating bed nets instead of antimalarial drugs is the objectively right answer, the same way buying a $500 TV instead of an identical TV that costs $5,000 is the right answer. And since saving a child from diarrheal disease costs $5,000, donating to an organization fighting malaria instead of an organization fighting diarrhea is the right answer, unless you are donating based on some criteria other than whether you're helping children or not.

Say all of the best Arctic explorers agree that the three most important things for surviving in the Arctic are good boots, a good coat, and good food. Perhaps they have run highly unethical studies in which they release thousands of people into the Arctic with different combination of gear, and consistently find that only the ones with good boots, coats, and food survive. Then there is only one best answer to the question "What gear do I buy if I want to survive" - good boots, good food, and a good coat. Your preferences are irrelevant; you may choose to go with alternate gear, but only if you don't mind dying.

And likewise, there is only one best charity: the one that helps the most people the greatest amount per dollar. This is vague, and it is up to you to decide whether a charity that raises forty children's marks by one letter grade for $100 helps people more or less than one that prevents one fatal case of tuberculosis per $100 or one that saves twenty acres of rainforest per $100. But you cannot abdicate the decision, or you risk ending up like the 11,000 people who accidentally decided that a pretty picture was worth more than a thousand people's lives.

Deciding which charity is the best is hard. It may be straightforward to say that one form of antimalarial therapy is more effective than another. But how do both compare to financing medical research that might or might not develop a "magic bullet" cure for malaria? Or financing development of a new kind of supercomputer that might speed up all medical research? There is no easy answer, but the question has to be asked.

What about just comparing charities on overhead costs, the one easy-to-find statistic that's universally applicable across all organizations? This solution is simple, elegant, and wrong. High overhead costs are only one possible failure mode for a charity. Consider again the Arctic explorer, trying to decide between a $200 parka and a $200 digital camera. Perhaps a parka only cost $100 to make and the manufacturer takes $100 profit, but the camera cost $200 to make and the manufacturer is selling it at cost. This speaks in favor of the moral qualities of the camera manufacturer, but given the choice the explorer should still buy the parka. The camera does something useless very efficiently, the parka does something vital inefficiently. A parka sold at cost would be best, but in its absence the explorer shouldn't hesitate to choose the the parka over the camera. The same applies to charity. An antimalarial net charity that saves one life per $500 with 50% overhead is better than an antidiarrheal drug charity that saves one life per $5000 with 0% overhead: $10,000 donated to the high-overhead charity will save ten lives; $10,000 to the lower-overhead will only save two. Here the right answer is to donate to the antimalarial charity while encouraging it to find ways to lower its overhead. In any case, examining the financial practices of a charity is helpful but not enough to answer the "which is the best charity?" question.

Just as there is only one best charity, there is only one best way to donate to that charity. Whether you volunteer versus donate money versus raise awareness is your own choice, but that choice has consequences. If a high-powered lawyer who makes $1,000 an hour chooses to take an hour off to help clean up litter on the beach, he's wasted the opportunity to work overtime that day, make $1,000, donate to a charity that will hire a hundred poor people for $10/hour to clean up litter, and end up with a hundred times more litter removed. If he went to the beach because he wanted the sunlight and the fresh air and the warm feeling of personally contributing to something, that's fine. If he actually wanted to help people by beautifying the beach, he's chosen an objectively wrong way to go about it. And if he wanted to help people, period, he's chosen a very wrong way to go about it, since that $1,000 could save two people from malaria. Unless the litter he removed is really worth more than two people's lives to him, he's erring even according to his own value system.

...and the same is true if his philanthropy leads him to work full-time at a nonprofit instead of going to law school to become a lawyer who makes $1,000 / hour in the first place. Unless it's one HELL of a nonprofit.

The Roman historian Sallust said of Cato "He preferred to be good, rather than to seem so". The lawyer who quits a high-powered law firm to work at a nonprofit organization certainly seems like a good person. But if we define "good" as helping people, then the lawyer who stays at his law firm but donates the profit to charity is taking Cato's path of maximizing how much good he does, rather than how good he looks.

And this dichotomy between being and seeming good applies not only to looking good to others, but to ourselves. When we donate to charity, one incentive is the warm glow of a job well done. A lawyer who spends his day picking up litter will feel a sense of personal connection to his sacrifice and relive the memory of how nice he is every time he and his friends return to that beach. A lawyer who works overtime and donates the money online to starving orphans in Romania may never get that same warm glow. But concern with a warm glow is, at root, concern about seeming good rather than being good - albeit seeming good to yourself rather than to others. There's nothing wrong with donating to charity as a form of entertainment if it's what you want - giving money to the Art Fund may well be a quicker way to give yourself a warm feeling than seeing a romantic comedy at the cinema - but charity given by people who genuinely want to be good and not just to feel that way requires more forethought.

It is important to be rational about charity for the same reason it is important to be rational about Arctic exploration: it requires the same awareness of opportunity costs and the same hard-headed commitment to investigating efficient use of resources, and it may well be a matter of life and death. Consider going to www.GiveWell.org and making use of the excellent resources on effective charity they have available.

Comment author: orthonormal 24 December 2010 09:14:25PM 0 points [-]

If I link this, and you later promote it to the front page, will my link go dead?

P.S. Why don't you just promote it now? Given the relative numbers involved, 'not signaling desire for karma' is a poor excuse.

Comment author: Yvain 24 December 2010 09:23:44PM *  0 points [-]

Oh. @#$*. I intended to post this to the front page. Now I can't figure out how to promote it.

Comment author: Yvain 23 December 2010 11:05:55PM 3 points [-]

Compare responses here to responses on the Reddit thread.

Comment author: komponisto 23 December 2010 08:50:56PM 9 points [-]

I've got to admit I disagree with a lot of Advancing Certainty

Do you actually disagree with any particular claim in Advancing Certainty, or does it just seem "off" to you in its emphasis? Because when I read your post, I felt myself "disagreeing" (and panicking at the rapid upvoting), but reflection revealed that I was really having something more like an ADBOC reaction. It felt to me that the intent of your post was to say "Boo confident probabilities!", while I tend to be on the side of "Yay confident probabilities!" -- not because I'm in favor of overconfidence, but rather because I think many worries about overconfidence here tend to be ill-founded (I suppose I'm something of a third-leveler on this issue.)

And indeed, when you see people complaining about overconfidence on LW, it's not usually because someone thinks that some political candidate has a 0.999999999 chance of winning an election; almost nobody here would think that a reasonable estimate. Instead, what you get is people saying that 0.0000000001 is too low a probability that God exists -- on the basis of nothing else than general worry about human overconfidence.

I think my anti-anti-overconfidence vigilance started when I realized I had been socially intimidated into backing off from my estimate of 0.001 in the Amanda Knox case, when in fact that was and remains an entirely reasonable number given my detailed knowledge of the case. The mistake I made was to present this number as if it were something that participants in my survey should have arrived at from a few minutes of reading. Those states -- the ones that survey participants were in, with reference classes like "highly controversial conviction with very plausible defense arguments" -- are what probabilities like 0.1 or 0.3 are for. My state, on the other hand, was more like "highly confident inside-view conclusion bolstered by LW survey results decisively on the same side of 50%".

But this isn't what the overconfidence-hawks argued. What they said, in essence, was that 0.001 was just somehow "inherently" too confident. Only "irrational" people wear the attire of "P(X) = 0.001"; We Here, by contrast, are Aware Of Biases Like Overconfidence, and only give Measured, Calm, Reasonable Probabilities.

That is the mistake I want to fight, now that I have the courage to do so. Though I can't find much to literally disagree about in your post, it unfortunately feels to me like ammunition for the enemy.

Comment author: Yvain 23 December 2010 09:13:14PM *  7 points [-]

I definitely did have the "ammunition for the enemy" feeling about your post, and the "belief attire" point is a good one, but I think the broad emotional disagreement does express itself in a few specific claims:

  1. Even if you were to control for getting tired and hungry and so on, even if you were to load your intelligence into a computer and have it do the hard work, I still don't think you could judge a thousand such trials and be wrong only once. I admit this may not be as real a disagreement as I'm thinking, because it may be a confusion on what sort of reference class we should use to pick trials for you.

  2. I think we might disagree on the Lord Kelvin claim. I think I would predict more of today's physical theories are wrong than you would.

  3. I think my probability that God exists would be several orders of magnitude higher than yours, even though I think you probably know about the same number of good arguments on the issue as I do.

Maybe our disagreement can be resolved empirically - if we were to do enough problems where we gave confidence levels on questions like "The area of Canada is greater than the area of the Mediterranean Sea" and use log odds scoring we might find one of us doing significantly better than the other - although we would have to do quite a few to close off my possible argument that we just didn't hit that one "black swan" question on which you'd say you're one in a million confident and then get it wrong. Would you agree that this would get to the heart of our disagreement, or do you think it revolves solely around more confusing philosophical questions?

(I took a test like that yesterday to test something and I came out overconfident, missing 2/10 questions at the 96% probability level. I don't know how that translates to more real-world questions and higher confidence levels, but it sure makes me reluctant to say I'm chronically underconfident)

Comment author: Yvain 23 December 2010 07:14:22PM *  10 points [-]

I'm really confused by this. Both examples sound like exactly what you should be doing.

If the Tibetan study was to determine how many people died in childbirth, and they refused to publish because not enough people were dying to support their preferred narrative, that would be bad. But the way you wrote it sounds like it was to determine differences in deaths across two conditions, but there weren't enough deaths to tell. In that case, stopping the study instead of publishing results that are probably false is the best possible course of action.

Likewise, doing studies in isolated villages is a good way to control for confounding variables. If you wanted to measure effectiveness of a cancer drug, you do it on people who are only taking the one drug, not people taking lots of drugs that might differ across groups. If you're measuring life expectancy in AIDS, you do it on patients who only have AIDS, not patients who have AIDS and also cancer. If a physicist is measuring the radioactivity of U-235, ey makes sure to use a purified sample of U-235, not a sample mixed with U-238. I don't even see what this has to do with double-blinding (which is about placebos and making sure subjects don't know what group they're in). This is just a basic attempt to isolate the experimental variable. I agree that given infinite time and resources they should try follow-up studies in less isolated villages to see if the results carry over, but in reality this seems like the best way to do things.

Am I misunderstanding the book's points?

Comment author: Kaj_Sotala 23 December 2010 12:18:35PM 0 points [-]

Could we have longer turn deadlines for a while, after this one has been processed? Christmas and all that.

Comment author: Yvain 23 December 2010 06:53:07PM *  0 points [-]

Sure. I'm going to be away a bit around New Years too, so that will take a while.

Comment author: Yvain 21 December 2010 04:56:35AM 0 points [-]

I am awaiting Austria's build order.

Comment author: JGWeissman 21 December 2010 12:07:48AM 3 points [-]

This problem contains correlated decision making, which is what makes copies anthropically confusing.

Comment author: Yvain 21 December 2010 03:12:50AM *  1 point [-]

Would it be the same problem if we said that there were nine people told they were potential deciders in the first branch, one person told ey was a potential decider in the second branch, and then we chose the decision of one potential decider at random (so that your decision had a 1/9 chance of being chosen in the first branch, but a 100% chance of being chosen in the second)? That goes some of the way to eliminating correlated decision making weirdness.

In response to Building Weirdtopia
Comment author: Alicorn 16 December 2010 04:19:32PM *  17 points [-]

Technological/Cognitive Weirdtopia: Everyone runs on computronium, in a simulation that starts out rather like normal, but everybody has an undo button: at your option you can undo everything except progress made in your own mind, up to any point in your life since the simulation began. There are safeguards in place to prevent two people from doing this at the exact same time, but otherwise there are no limitations on use; you can redo a second or a century, once or a thousand times. It takes a lot of "real" time for the simulation to progress to everyone's satisfaction beyond the first five minutes.

Comment author: Yvain 18 December 2010 07:56:34PM *  1 point [-]

See: The Void Trilogy, Peter Hamilton.

Comment author: Carinthium 15 December 2010 04:36:27PM 0 points [-]

I'm screwed anyway, and the idea of checking on my rival's moves is not appealing. Resigning.

Comment author: Yvain 18 December 2010 05:43:36PM *  3 points [-]

In the interests of reducing uncertainty for the other players: I officially accept Italy's resignation. Italy has not sent in any orders for Fall 1905, and any future orders sent in by Italy will be disregarded.

Italy is now under civil disorder rules. All Italian armies will hold position and not support one another. If Italy loses supply centers, it will disband armies furthest away from its capital first.

As you may have noticed, I have finished my travel for this weekend a bit early and will update the turn as soon as I get everyone's orders.

Comment author: komponisto 16 December 2010 06:44:44PM *  5 points [-]

But I also feel that if I tried to use an argument at least that subtle, when applied to something I am at least as confused about as how ontologically complex a first cause should be, to disprove things at least as widely believed as religion, a million times, I would be wrong at least once.

See Advancing Certainty. The fact that this statement sounds comfortably modest does not exempt it from the scrutiny of the Fundamental Question of Rationality (why do you believe what you believe?). I respectfully submit that if the answer is "because I have been wrong before, where I was equally confident, in previous eras of my life when I wasn't using arguments this powerful (they just felt powerful to me at the time)", that doesn't suffice -- for the same reason that the Lord Kelvin argument doesn't suffice to show that arguments from physics can't be trusted (unless you don't think physics has learned anything since Kelvin).

Comment author: Yvain 18 December 2010 04:38:37PM *  8 points [-]

I've got to admit I disagree with a lot of Advancing Certainty. The proper reference class for a modern physicist who is well acquainted with the mistakes of Lord Kelvin and won't do them again is "past scientists who were well acquainted with the mistakes of their predecessors and plan not to do them again", which I imagine has less than a hundred percent success rate and which might have included Kelvin.

It would be a useful exercise to see whether the most rational physicists of 1950 have more successful predictions as of 2000 than the most rational physicists of 1850 did as of 1900. It wouldn't surprise me if this were true, and so, then the physicists of 2000 could justly put themselves in a new reference class and guess they will be even more successful as of 2050 than the 1950ers were in 2000. But if the success rate after fifty years remains constant, I wouldn't want to say "Yeah, well , we've probably solved all those problems now, so we'll do better".

Comment author: Eliezer_Yudkowsky 16 December 2010 05:09:44PM 3 points [-]

I should like to know Yvain's prior on this.

Comment author: Yvain 18 December 2010 04:25:51PM *  0 points [-]

On the "abracadabra" example? The overwhelming majority would come from the possibility that any time anything whatsoever happens the world is "destroyed", for some weird, maybe anthropic use of the word "destroyed" I don't understand compatible with me still being here.

If we limit it to "abracadabra" and nothing else, that's complex enough that < 1/trillion just picking it out of hypothesis space (lots of combinations of sounds that could destroy the world, lots of things that aren't combinations of sounds).

Comment author: [deleted] 16 December 2010 04:05:26PM *  1 point [-]

We have hypothesis H and evidence E, and we dutifully compute

P(H) * P(E | H) / P(E)

It sounds like your advice is: don't update yet! Especially if this number is very small. We might have made a mistake. But then how should we update? "Round up" seems problematic.

Comment author: Yvain 16 December 2010 04:39:24PM *  6 points [-]

I'm not saying not to use Bayes' theorem, I'm saying to consider very carefully what to plug into "E". In the election example, your evidence is "A guy on a website said that there was a 999,999,999 in a billion chance that the incumbent would win." You need to compute the probability of the incumbent winning given this actual evidence (the evidence that a guy on a website said something), not given the evidence that there really is a 999,999,999/billion chance. In the cosmic ray example, your evidence would be "There's an argument that looks like it should make a less than 10^20 chance of apocalypse", which may have different evidence value depending on how well your brain judges the way arguments look.

EDIT: Or what nerzhin said.

Comment author: Drahflow 16 December 2010 09:07:59AM 1 point [-]

But it's hard for me to be properly outraged about this, because the conclusion that the LHC will not destroy the world is correct.

What is your argument for claiming that the LHC will not destroy the world?

That the world still exists albeit ongoing experiments is easily explained by the fact that we are necessarily living in those branches of the universe where the LHC didn't destroy the world. (On an related side note: Has the great filter been found yet?)

Comment author: Yvain 16 December 2010 02:26:32PM 1 point [-]

Good point. I've changed this to "since the LHC did not destroy the world", which is true regardless of whether it destroyed other branches.

Comment author: Johnicholas 16 December 2010 01:12:14PM 5 points [-]

First, great post. Second, general injunctions against giving very low probabilities to things seems to be taken by many casual readers as endorsements of the (bad) behavior "privilege the hypothesis" - e.g. moving the probability from very small to moderately small that God exists. That's not right, but I don't have excellent arguments for why it's not right. I'd love it if you wrote an article on choosing good priors.

Cosma Shalizi has done some technical work that seems (to my incompetent eye) to be relevant:

http://projecteuclid.org/DPubS?verb=Display&version=1.0&service=UI&handle=euclid.ejs/1256822130&page=record

That is, he takes Bayesian updating, which requires modeling the world, and answers the question 'when would it be okay to use Bayesian updating, even though we know the model is definitely wrong - e.g. too simple?'. (Of course, making your model "not obviously wrong" by adding complexity isn't a solution.)

Comment author: Yvain 16 December 2010 02:23:59PM 14 points [-]

I am still confused about how small the probability I should use in the God question is. I understand the argument about privileging the hypothesis and about intelligent beings being very complex and fantastically unlikely.

But I also feel that if I tried to use an argument at least that subtle, when applied to something I am at least as confused about as how ontologically complex a first cause should be, to disprove things at least as widely believed as religion, a million times, I would be wrong at least once.

Confidence levels inside and outside an argument

129 Post author: Yvain 16 December 2010 03:06AM

Related to: Infinite Certainty

Suppose the people at FiveThirtyEight have created a model to predict the results of an important election. After crunching poll data, area demographics, and all the usual things one crunches in such a situation, their model returns a greater than 999,999,999 in a billion chance that the incumbent wins the election. Suppose further that the results of this model are your only data and you know nothing else about the election. What is your confidence level that the incumbent wins the election?

Mine would be significantly less than 999,999,999 in a billion.

When an argument gives a probability of 999,999,999 in a billion for an event, then probably the majority of the probability of the event is no longer in "But that still leaves a one in a billion chance, right?". The majority of the probability is in "That argument is flawed". Even if you have no particular reason to believe the argument is flawed, the background chance of an argument being flawed is still greater than one in a billion.


More than one in a billion times a political scientist writes a model, ey will get completely confused and write something with no relation to reality. More than one in a billion times a programmer writes a program to crunch political statistics, there will be a bug that completely invalidates the results. More than one in a billion times a staffer at a website publishes the results of a political calculation online, ey will accidentally switch which candidate goes with which chance of winning.

So one must distinguish between levels of confidence internal and external to a specific model or argument. Here the model's internal level of confidence is 999,999,999/billion. But my external level of confidence should be lower, even if the model is my only evidence, by an amount proportional to my trust in the model.



Is That Really True?

One might be tempted to respond "But there's an equal chance that the false model is too high, versus that it is too low." Maybe there was a bug in the computer program, but it prevented it from giving the incumbent's real chances of 999,999,999,999 out of a trillion.

The prior probability of a candidate winning an election is 50%1. We need information to push us away from this probability in either direction. To push significantly away from this probability, we need strong information. Any weakness in the information weakens its ability to push away from the prior. If there's a flaw in FiveThirtyEight's model, that takes us away from their probability of 999,999,999 in of a billion, and back closer to the prior probability of 50%

We can confirm this with a quick sanity check. Suppose we know nothing about the election (ie we still think it's 50-50) until an insane person reports a hallucination that an angel has declared the incumbent to have a 999,999,999/billion chance. We would not be tempted to accept this figure on the grounds that it is equally likely to be too high as too low.

A second objection covers situations such as a lottery. I would like to say the chance that Bob wins a lottery with one billion players is 1/1 billion. Do I have to adjust this upward to cover the possibility that my model for how lotteries work is somehow flawed? No. Even if I am misunderstanding the lottery, I have not departed from my prior. Here, new information really does have an equal chance of going against Bob as of going in his favor. For example, the lottery may be fixed (meaning my original model of how to determine lottery winners is fatally flawed), but there is no greater reason to believe it is fixed in favor of Bob than anyone else.2

Spotted in the Wild

The recent Pascal's Mugging thread spawned a discussion of the Large Hadron Collider destroying the universe, which also got continued on an older LHC thread from a few years ago. Everyone involved agreed the chances of the LHC destroying the world were less than one in a million, but several people gave extraordinarily low chances based on cosmic ray collisions. The argument was that since cosmic rays have been performing particle collisions similar to the LHC's zillions of times per year, the chance that the LHC will destroy the world is either literally zero, or else a number related to the probability that there's some chance of a cosmic ray destroying the world so miniscule that it hasn't gotten actualized in zillions of cosmic ray collisions. Of the commenters mentioning this argument, one gave a probability of 1/3*10^22, another suggested 1/10^25, both of which may be good numbers for the internal confidence of this argument.

But the connection between this argument and the general LHC argument flows through statements like "collisions produced by cosmic rays will be exactly like those produced by the LHC", "our understanding of the properties of cosmic rays is largely correct", and "I'm not high on drugs right now, staring at a package of M&Ms and mistaking it for a really intelligent argument that bears on the LHC question", all of which are probably more likely than 1/10^20. So instead of saying "the probability of an LHC apocalypse is now 1/10^20", say "I have an argument that has an internal probability of an LHC apocalypse as 1/10^20, which lowers my probability a bit depending on how much I trust that argument".

In fact, the argument has a potential flaw: according to Giddings and Mangano, the physicists officially tasked with investigating LHC risks, black holes from cosmic rays might have enough momentum to fly through Earth without harming it, and black holes from the LHC might not3. This was predictable: this was a simple argument in a complex area trying to prove a negative, and it would have been presumptous to believe with greater than 99% probability that it was flawless. If you can only give 99% probability to the argument being sound, then it can only reduce your probability in the conclusion by a factor of a hundred, not a factor of 10^20.

But it's hard for me to be properly outraged about this, since the LHC did not destroy the world. A better example might be the following, taken from an online discussion of creationism4 and apparently based off of something by Fred Hoyle:

In order for a single cell to live, all of the parts of the cell must be assembled before life starts. This involves 60,000 proteins that are assembled in roughly 100 different combinations. The probability that these complex groupings of proteins could have happened just by chance is extremely small. It is about 1 chance in 10 to the 4,478,296 power. The probability of a living cell being assembled just by chance is so small, that you may as well consider it to be impossible. This means that the probability that the living cell is created by an intelligent creator, that designed it, is extremely large. The probability that God created the living cell is 10 to the 4,478,296 power to 1.

Note that someone just gave a confidence level of 10^4478296 to one and was wrong. This is the sort of thing that should never ever happen. This is possibly the most wrong anyone has ever been.

It is hard to say in words exactly how wrong this is. Saying "This person would be willing to bet the entire world GDP for a thousand years if evolution were true against a one in one million chance of receiving a single penny if creationism were true" doesn't even begin to cover it: a mere 1/10^25 would suffice there. Saying "This person believes he could make one statement about an issue as difficult as the origin of cellular life per Planck interval, every Planck interval from the Big Bang to the present day, and not be wrong even once" only brings us to 1/10^61 or so. If the chance of getting Ganser's Syndrome, the extraordinarily rare psychiatric condition that manifests in a compulsion to say false statements, is one in a hundred million, and the world's top hundred thousand biologists all agree that evolution is true, then this person should preferentially believe it is more likely that all hundred thousand have simultaneously come down with Ganser's Syndrome than that they are doing good biology5

This creationist's flaw wasn't mathematical; the math probably does return that number. The flaw was confusing the internal probability (that complex life would form completely at random in a way that can be represented with this particular algorithm) with the external probability (that life could form without God). He should have added a term representing the chance that his knockdown argument just didn't apply.

Finally, consider the question of whether you can assign 100% certainty to a mathematical theorem for which a proof exists. Eliezer has already examined this issue and come out against it (citing as an example this story of Peter de Blanc's). In fact, this is just the specific case of differentiating internal versus external probability when internal probability is equal to 100%. Now your probability that the theorem is false is entirely based on the probability that you've made some mistake.

The many mathematical proofs that were later overturned provide practical justification for this mindset.

This is not a fully general argument against giving very high levels of confidence: very complex situations and situations with many exclusive possible outcomes (like the lottery example) may still make it to the 1/10^20 level, albeit probably not the 1/10^4478296. But in other sorts of cases, giving a very high level of confidence requires a check that you're not confusing the probability inside one argument with the probability of the question as a whole.

Footnotes

1. Although technically we know we're talking about an incumbent, who typically has a much higher chance, around 90% in Congress.

2. A particularly devious objection might be "What if the lottery commissioner, in a fit of political correctness, decides that "everyone is a winner" and splits the jackpot a billion ways? If this would satisfy your criteria for "winning the lottery", then this mere possibility should indeed move your probability upward. In fact, since there is probably greater than a one in one billion chance of this happening, the majority of your probability for Bob winning the lottery should concentrate here!

3. Giddings and Mangano then go on to re-prove the original "won't cause an apocalypse" argument using a more complicated method involving white dwarf stars.

4. While searching creationist websites for the half-remembered argument I was looking for, I found what may be my new favorite quote: "Mathematicians generally agree that, statistically, any odds beyond 1 in 10 to the 50th have a zero probability of ever happening." 

5. I'm a little worried that five years from now I'll see this quoted on some creationist website as an actual argument.

Comment author: Yvain 15 December 2010 05:12:03PM *  11 points [-]

The RTPJ doesn't control morality, it controls theory of mind. This is known and the study admits it (though the BBC doesn't).

All this shows is that moral judgments that require considering people's mental states call the theory of mind function, which is predictable. If you can't think about someone's beliefs at all, you can't use them as a mitigating factor in a moral judgment.

If you gave someone a stroke and caused hemineglect, they'd stop condemning immoral actions that happened on one side of the visual field, but that wouldn't mean you'd discovered something interesting about their morality.

The study gets this exactly right: "Our hypothesis therefore is that TMS to the RTPJ affects an input to moral judgment (i.e., belief information) but not the process of moral judgment per se."

Comment author: Carinthium 15 December 2010 04:36:27PM 0 points [-]

I'm screwed anyway, and the idea of checking on my rival's moves is not appealing. Resigning.

Comment author: Yvain 15 December 2010 04:52:47PM 1 point [-]

I only meant by this that you should make sure I didn't make a mistake that hurts you and helps your rivals. You don't have to if you don't want to.

Comment author: Jack 15 December 2010 04:24:12PM 0 points [-]

The second map reports the actions of the previously turn, not the positions for the new turn as it usually does.

Comment author: Yvain 15 December 2010 04:26:39PM 1 point [-]

Fixed.

Comment author: alethiophile 14 December 2010 06:44:17AM 4 points [-]

I would say that while there are hypotheses with such low priors as to make it irrational to expend the effort to check them (and Harry would probably have assigned such a low probability to the existence of HP-style magic, and magical Britain, before he got the letter), one of these hypotheses being promoted specifically to your attention in the manner of the letter probably raises the prior to the point where it's worth testing, at least.

Comment author: Yvain 14 December 2010 09:24:23PM *  24 points [-]

The probability of magic is still vanishingly vanishingly low, but given how useful magic would be it might still be worth Harry's time to test for it.

I hereby dub this class of argument Pascal's Muggle

Comment author: Perplexed 14 December 2010 04:03:23PM 2 points [-]

I hesitate to mention this now, but doesn't Italy also have the option of disbanding, rather than retreating? And if the army disbands, doesn't Italy get to build - perhaps in Rome?

Comment author: Yvain 14 December 2010 04:07:44PM *  2 points [-]

@#$!$#@. By which I mean yes. Okay, I will just wait for Italy to do something and then figure out what happens from there.

Comment author: Jack 13 December 2010 09:29:39PM *  5 points [-]

I've noticed for a while how quote-worthy the comments here are. It might be worthwhile to have those quote-worthy bits all in one place. Maybe a "Most Quotable Comments" thread? Yes, this is a little bit self-indulgent.

Comment author: Yvain 14 December 2010 12:43:28PM 4 points [-]

Why not remove the "no LWers on Rationality Quotes" restriction and let the problem take care of itself?

Comment author: Emile 13 December 2010 08:51:37PM 2 points [-]

Agree, agree, but ...

fewer gimmicks

... what do you mean by gimmicks, here?

Comment author: Yvain 14 December 2010 12:19:34PM 0 points [-]

I don't want to say anything specific enough to offend anyone, but some of the ideas I saw on the "neat stuff" thread seemed sort of gimmicky to me.

Comment author: Jack 14 December 2010 10:50:33AM *  1 point [-]

Uhhhhh, I don't think Italy is allowed to issue that retreat order! Since he was already bounced from Rome his choices are Piedmont and Tuscany. Jdip won't accept other imputs.

Comment author: Yvain 14 December 2010 12:01:48PM *  1 point [-]

Darnit, quick rule check says you're right.

Okay, I want an Italian retreat order, and new Italian and Austrian orders for this turn AS SOON AS POSSIBLE. If Austria wants to send me two new move orders (one contingent upon retreat to Tus, one on retreat to Pie) so I'm only waiting for Italy, that's fine too. Anyone else may also send me changed orders in the "conditional upon retreat to Piedmont/Tuscany" format if they think it's important.

Sorry for the inconvenience.

Comment author: Yvain 14 December 2010 07:56:10AM *  2 points [-]

$@!%!#$%!#!@!

As Austria recently pointed out, the last turn should have had Apulia moving to Rome, which means that it bounced with Italy and Italy did not move into Rome. That means Italy is still in Venice, which means it needs to issue a retreat order.

EDIT: Italians can't retreat into Rome. Italy, please send retreat order and revised move orders. Austria, please send revised move orders.

I know this turn is late. I didn't get Italy's orders until several hours after the deadline. Sorry.

Comment author: Yvain 13 December 2010 08:48:55PM *  27 points [-]

I'd like to see less about the application of rationality to various different everyday life skills (which usually end up being bad self-help), less navel-gazing about how great we are and how great rationality is, fewer gimmicks, and more attempt to dissolve confusions. explain previously puzzling things, and point out logical pitfalls that one might otherwise fall into. This is what Eliezer did so well in the Sequences, and I can't believe that there aren't any good problems left to cover.

Comment author: Yvain 12 December 2010 10:12:52PM 7 points [-]

Okay, I admit it.

I've PMed jsalvatier with information on how to pay.

Some people said my essay wasn't polished, or should have more internal LW links. These were both fair points. If someone is going to do something with these essays, please feel free to tell me how to make it better.

Comment author: Yvain 12 December 2010 01:03:46PM *  7 points [-]

The Moral Animal by Robert Wright.

EDIT: The comment above made a good point. Do you mean "conscience" or "consciousness"? Moral Animal is good for conscience, not so good for consciousness.

In response to Calling LW Londoners
Comment author: Yvain 12 December 2010 01:00:12PM *  0 points [-]

I'll be in London from 30/12 to 3/1, but with several other commitments. Any time in that period would be possible but 2/1 would be best. But I've already messed up enough other meets with my scheduling issues, so don't make too much effort to work around me unless those times are also great for everyone else.

I also would like to go to London in February or March sometime, so if you plan it on a weekend in one of those two months I could probably make it with enough (at least two or three weeks') notice.

Comment author: Kingreaper 11 December 2010 07:49:01PM 2 points [-]

And that because the experiment has been tried before millions of times in the form of cosmic rays

Not millions of times. Not even just billions of times.

From a back of the envelope calculation they've been tried >10^16 times a year.

For the past 10^9 years.

That's 10^25 times

And that's probably several orders of magnitude low.

So yes, treating it as something with a non-zero probability of destroying the planet is silly.

Especially because every model I've seen that says it'd destroy the planet would also have it destroy the sun. Which has 10^4 times the surface area of the Earth, and would have correspondingly more cosmic ray collisions.

Comment author: Yvain 11 December 2010 07:58:35PM *  6 points [-]

Read page 848 of http://arxiv.org/ftp/arxiv/papers/0912/0912.5480.pdf

I'm guessing you weren't aware of all the technical intricacies of this argument nor the necessity of bringing in white dwarf stars to clinch it. Now, it turns out you got lucky, because white dwarf stars do end out clinching the argument. But if there's a facet of the argument you don't understand, or there's even a tiny possibility there's a facet of the argument you don't fully understand, you don't go saying there's zero probability.

Comment author: David_Gerard 11 December 2010 07:22:42PM *  -1 points [-]

Again, let's say by "but there's still a chance" you're saying the chance of CERN causing an apocalypse scenario is less than one in a billion.

No, it's a much smaller order of number than that. You're still starting from "but there's a chance, right?"

The rest of your post is reasoning from your own ignorance of the specific topic of the LHC, but not from that of everyone else. You appear not to have grasped the point of what I just wrote. Please echo back your understanding of what I wrote.

Comment author: Yvain 11 December 2010 07:28:36PM *  1 point [-]

I understand you as saying that cosmic ray collisions that happen all the time are very similar to the sort of collisions at CERN, and since they don't cause apocalypses, CERN won't either. And that because the experiment has been tried before millions of times in the form of cosmic rays, this "CERN won't either" isn't on the order of "one in a million" or "one in a billion" but is so vanishingly small that it would be silly to even put a number to it.

Tell me if I understood you correctly and if I did I will try to rephrase my post and my objections to what you said so they are more understandable.

Comment author: David_Gerard 11 December 2010 06:49:06PM *  0 points [-]

In the case of your original assertion - that it was reasonable to worry about the risks of the LHC - the argument for the probability of disaster being too small to worry about is that we're not working out the probability assuming such events have never happened before - we're working out the probability assuming such events and stronger ones happen all the time, because they do. So very many collisions occur just near Earth of greater energies that this puts a strong upper bound on the chances of disaster occurring in the LHC itself. Even multiplied by 6E9, the number is, as I said, much less like 1E-6 and much more like "but there's still a chance, right?"

No. No, there really isn't.

Comment author: Yvain 11 December 2010 07:19:47PM *  1 point [-]

Again, let's say by "but there's still a chance" you're saying the chance of CERN causing an apocalypse scenario is less than one in a billion. You say that "the argument" for this is that such collisions happen near Earth all the time.

Suppose I were to posit that black holes produced by cosmic rays have an acceleration that would lead them to fly through the Earth without harming it, but black holes produced by the LHC would be slower and thus able to destroy the Earth where their cosmic-ray-produced brethren could not.

Suppose I were to tell you that either this above paragraph was the view of a significant part of the relevant physicist community (say, greater than five percent) or that I was bluffing and totally made it up.

I offer you a bet. If I'm bluffing, I'll give you a dollar. If I'm not, you give me ten thousand dollars. No, you can't Google it to check. If your utility function isn't linear with respect to money, I'm happy to lower it to something like 1:1000 instead.

If you don't take the bet, it means you're not even sure to ten thousand to one odds that that particular argument holds, which makes it very iffy to use as the lynchpin of an argument for billion to one odds.

Comment author: Vaniver 11 December 2010 04:07:49PM 3 points [-]

Most of life is structured as a negative lottery. You get in a car, you get where you're going much faster- but if the roulette ball lands on 00, you're in the hospital or dead. (If it only lands on 0, then you're just facing lost time and property.)

And so some people are mildly afraid of cars, but mostly people are just afraid of bad driving or not being in control- the negative lottery aspect of cars is just a fact of life, taken for granted and generally ignored when you turn the key.

Comment author: Yvain 11 December 2010 05:51:18PM 5 points [-]

The reason I recommend David not play the inverse lottery isn't because all things that give small rewards for a small probability of great loss are bad, it's because the inverse lottery (like the regular lottery) is set up so that the expected utility of playing is lower than the expected utility of not playing. An inverse lottery in which the expected utility of playing is better than the expected utility of not playing would be a good bet.

A good argument for driving cars wouldn't be that an accident could never happen and is ridiculous (which is how I interpret David's pro-LHC argument) but that the benefits gained from driving cars outweigh the costs.

Spring 1912: A New Heaven And A New Earth

18 Yvain 13 November 2010 05:11PM

And so it came to pass that on Christmas Day 1911, the three Great Powers of Europe signed a treaty to divide the continent between them peacefully, ending what future historians would call the Great War.

The sun truly never sets on King Jack's British Empire, which stretches from Spain to Stockholm, from Casablanca to Copenhagen, from the fringes of the Sahara to the coast of the Arctic Ocean. They rule fourteen major world capitals, and innumerable smaller towns and cities, the greatest power of the age and the unquestioned master of Western Europe.

From the steppes of Siberia to the minarets of Istanbul, the Ottoman Empire is no longer the Sick Man of Europe but stands healthy and renewed, a colossus every bit the equal of the Christian powers to its west. Its Sultan calls himself the Caliph, for the entire Islamic world basks in his glory, and his Grand Vizier has been rewarded with a reputation as one of the most brilliant and devious politicians of the age. At his feet grovel representatives of twelve great cities, and even far-flung Tunis has not escaped his sway.

And in between, the Austro-Hungarian Empire straddles the Alps and ancient Italy. Its lack of natural borders presented no difficulty for its wily Emperor, who successfully staved off the surrounding powers and played his enemies off against one another while building alliances that stood the test of time. Eight great cities pay homage to his double-crown, and he is what his predecessors could only dream of being - a true Holy Roman Emperor.

And hidden beneath the tricolor map every student learns in grammar school are echoes of subtler hues. In Germany, people still talk of the mighty Kajser Sotala I, who conquered the ancient French enemy and extended German rule all the way to the Mediterranean, and they still seeth and curse at his dastardly betrayal by his English friends. In Russia, Princess Anastasia claims to be the daughter of Czar Perplexed, and recounts to everyone who will listen the story of her stoic father, who remained brave until the very end; at her side travels a strange bearded man who many say looks like Rasputin, the Czar's long-missing adviser. The French remember President Andreassen, who held off the combined armies of England and Germany for half a decade, and many still go on pilgrimage to Liverpool, the site of their last great victory. And in Italy, Duke Carinthium has gone down in history beside Tiberius and Cesare Borgia as one of their land's most colorful and fascinating leaders.

And the priests say that the same moment the peace treaty was signed, the blood changed back to water, and the famines ended, and rain fell in the lands parched by drought. Charles Taze Russell, who had been locked in his room awaiting the Apocalypse, suddenly ran forth into the midwinter sun, shouting "Our doom has been lifted! God has granted us a second chance!" And the mysterious rectangular wall of force separating Europe from the rest of the world blinked out of existence.

Pope Franz I, the new Austrian-supported Pontiff in Rome, declares a month of thanksgiving and celebration. For, he says, God has tested the Europeans for their warlike ways, isolating them from the rest of the earth lest their sprawling empires plunge the entire planet into a world war that might kill millions. Now, the nobility of Europe finally realizing the value of peace, the curse has been lifted, and the empires of Europe can once more interact upon the world stage.

Chastened by their brush with doom, yet humbled by the lesson they had been given, the powers of Europe send missionaries through the dimensional portal, to convince other worlds to abandon their warlike ways and seek universal brotherhood. And so history ends, with three great powers living together side by side and striving together for a better future and a positive singularity.

...

On to the more practical parts. If you think you've learned lessons this game worth telling the rest of Less Wrong, you should send them to either myself or Jack. I say either myself or Jack because Jack had the most supply centers and therefore deserves some karma which he could most easily get by posting the thread which the other two winners then comment on, or if you insist that three way tie means three way tie, I'll post the thread and the three winners can all comment and get up-voted. We'll talk about it in the comments.

Thanks to everyone who played in this game. I was very impressed - it's one of the rare games I have moderated that hasn't been ruined by people constantly forgetting to send orders, or people ragequitting when things don't go their way, or people being totally incompetent and throwing the game to the first person to declare war on them, or any of the other ways a Diplomacy game can go wrong. Everyone fought hard and well and honorably (for definitions of honor compatible with playing Diplomacy). It was a pleasure to serve as your General Secretary.

 

All previous posts and maps from this game are archived. See this comment for an explanation of how to access the archives.

Diplomacy as a Game Theory Laboratory

44 Post author: Yvain 12 November 2010 10:19PM

Game theory. You've studied the posts, you've laughed at the comics, you've heard the music1. But the best way to make it Truly Part Of You is to play a genuine game, and I have yet to find any more effective than Diplomacy.

 

Diplomacy is a board game for seven people played on a map of WWI Europe. The goal is to capture as many strategic provinces ("supply centers") as possible; eighteen are needed to win. But each player's country starts off with the same sized army, and there is no luck or opportunity for especially clever tactics. The most common way to defeat an enemy is to form coalitions with other players. But your enemies will also be trying to form coalitions, and the most profitable move is often to be a "double agent", stringing both countries along as long as you can. All game moves are written in secret and revealed at the same time and there are no enforcement mechanisms, so alliances, despite their central importance, aren't always worth the paper they're printed on.

 

The conditions of Diplomacy - competition for scarce resources, rational self-interested actors, importance of coalitions, lack of external enforcement mechanisms - mirror the conditions of game theoretic situations like the Prisoner's Dilemma (and the conditions of most of human evolution!) and so make a surprisingly powerful laboratory for analyzing concepts like trust, friendship, government, and even religion.

 

Over the past few months, I've played two online games of Diplomacy. One I won through a particularly interesting method; the other I lost quite badly, but with an unusual consolation. This post is based on notes I took during the games about relevant game theoretic situations. You don't need to know the rules of Diplomacy to understand the post, but if you want a look you can find them here.

 

Study One: The Prisoner's Dilemma


The Prisoner's Dilemma is a classic case in game theory in which two players must decide whether or not to cooperate for a common goal. If both players cooperate, they both do better than if both defect, but one player can win big by defecting when the other cooperates. This situation is at the heart of Diplomacy.

 

Germany and France have agreed to ally against Britain. Both countries have demilitarized their mutual border, and are concentrating all of their forces to the north, where they take province after province of British territory.

 

But Britain is fighting back; not successfully, but every inch of territory is hard-won. France is doing well for itself and has captured a few British cities, but it could be doing better. The French player thinks to eirself: I could either continue battering against the heavily defended British lines, or I could secretly ally with Britain, stab Germany in the back, and waltz in along our undefended mutual border before the Germans even know what hit them. Instead of fighting for each inch of British land, I could be having dinner in Berlin within a week.

 

Meanwhile, in Berlin, the German player is looking towards France's temptingly undefended border and thinking the exact same thing.

 

If both France and Germany are honorable, and if both countries know the other is honorable, the two of them can continue fighting Britain with a two-to-one numerical advantage and probably divide England's lucrative territory among the two of them.

 

If Germany is naively trusting and France is a dishonest backstabber, then France can get obscene rewards by rolling over Germany while the Kaiser's armies are tied up on the fields of England.

 

If both countries are suspicious of the other, or if both countries try to backstab each other simultaneously, then they will both divert forces away from the war on England to guard their mutual border. They will not gain any territory in England, and they will not gain any territory along their border. They've not only stabbed each other in the back, they've shot themselves in the foot.

 

Study Two: Parfit's Hitch-Hiker


The wiki describes Derek Parfit's famous hitchhiker problem as:

 

Suppose you're out in the desert, running out of water, and soon to die - when someone in a motor vehicle drives up next to you. Furthermore, the driver of the motor vehicle is a perfectly selfish ideal game-theoretic agent, and even further, so are you; and what's more, the driver is Paul Ekman, who's really, really good at reading facial microexpressions. The driver says, "Well, I'll convey you to town if it's in my interest to do so - so will you give me $100 from an ATM when we reach town?"

 

Now of course you wish you could answer "Yes", but as an ideal game theorist yourself, you realize that, once you actually reachtown, you'll have no further motive to pay off the driver. "Yes," you say. "You're lying," says the driver, and drives off leaving you to die.

 

The so-called Key Lepanto opening is one of the more interesting opening strategies in Diplomacy, and one that requires guts of steel to pull off. It goes like this: Italy and Austria decide to ally against Turkey. This is common enough, and hindered by the fact that Turkey is probably expecting it and Italy's kind of far away from Turkey anyway.

 

So Italy and Austria do something unexpected. Italy swears loudly and publicly that ey's allied with Austria. Then, the first turn, Italy moves deep into undefended Austrian territory! Austria is incensed, and curses loud and long at Italy's betrayal and at eir own stupidity for leaving the frontier unguarded. Turkey laughs and leaves the two of them to their war when - boom - Austria and Italy launch a coordinated attack against Turkey from Italy's base deep in Austrian territory. The confused Turkey has no chance to organize a resistance before combined Italo-Austrian forces take Constantinople.

 

It's frequently a successful strategy, especially for Italy. You know what else is a successful strategy for Italy? Doing this up to the point where they take over lots of Austrian territory, forgetting the part where it was all just a ploy, and then ending up in control of lots of Austrian territory, after which they can fight Turkey at their leisure.

 

It's very much in Italy's advantage to play a Key Lepanto opening, and they may beg the Austrian player to go for it, saying correctly that it would benefit both of them. But the Austrian player very often refuses, telling Italy that ey would have no incentive not to just keep the conquered territory.

 

This problem resembles the Hitchhiker: Italy is the lost man, and Austria is the driver. Italy really wants Austria to help em play the awesome Key Lepanto opening, but Austria knows that ey would have no incentive not to break his promise once Austria's given him the help he needs. As a result, neither country gets what they want. The Key Lepanto opening is played only rarely, and this is one of the reasons.

 

Study Three: Enforceable Side Contracts


The Prisoner's Dilemma is nontrivial because there's no enforcement mechanism. In the presence of an enforcement mechanism, it becomes much simpler. Say two mobsters are about to be arrested, and expect to be put in a Prisoner's Dilemma type situation. They approach the mob boss with a contract with both of their names on it, saying that they have both agreed that if either of them testifies against the other, the mob boss should send his goons to shoot the rat.

 

For many payoff matrices, signing this contract will be a no-brainer. It ensures your opponent will cooperate at the relatively low cost of forcing you to cooperate yourself, and almost guarantees you safe passage into the desirable (C,C) square. Not only does it prevent your opponent doesn't defect out of sheer greed, but it prevents your opponent from worrying that you're going to defect and then defecting emself to save emself from being the chump.

 

The game of Diplomacy I won, I won through an enforceable side contract (which lost me a friend and got me some accusations of cheating, but this is par for the course for a good Diplomacy game). I was Britain; my friend H was France. H and I knew each other from an medieval times role-playing game, in which we both held land and money. The medieval kingdom of this game had a law on the books that any oath witnessed by a noble was binding on both parties and would be enforced by the king. So H and I went into our role-playing game and swore an oath before a cooperative noble, declaring that we would both aid each other in a permanent alliance in Diplomacy, or else all our in-game lands and titles would be forfeit.

 

A lot of people made fun of me for this, including H, but in my defense I did end up winning the game. H and I were able to do things that would otherwise have been impossible; for example, in order to convince our enemy Germany that we were at war, I took over the French city of Brest. Normally, this would be almost impossible for two allies to coordinate, even as a red herring, for exactly the reasons listed in the Hitchhiker problem above. Since the two of us were able to trust each other absolutely, this otherwise difficult maneuver became easy.

 

One of the advantages to strong central government is that it provides an enforcement mechanism for contracts, which benefits all parties.

 

Study Four: Religion As Enforcement


Religion is a special case of the enforceable side-contract in which God is doing the enforcing. God doesn't have to exist for this to work; as long as at least one party believes He does, the threat of punishment will be credible. The advantage of being able to easily make enforceable side contracts even in the absence of social authority may be one reason religion became so popular, and if humans do turn out to have a genetic tendency toward belief, the side contracts might have provided part of the survival advantage that spread the gene.

 

In a Youngstown Variant game (like Diplomacy, but with Eurasia instead of just Europe), I was playing Italy and after colonizing Africa was trying to juggle my forces around to defend borders with Germany, France, Turkey, and India.

 

India was played by my friend A, who I sometimes have philosophical discussions with and who I knew to be an arch-conservative religion-and-family-values type. I decided to try something which, as far as I know, no one's ever tried in a Diplomacy game before. "Do you swear in the name of God and your sacred honor that you won't attack me?" I asked.

 

"Yes," said A, and I knew he meant it, because he takes that sort of thing really seriously. I don't know if he thought he would literally go to Hell if he broke his oath, but I'm pretty sure he wasn't willing to risk it over a board game. So I demilitarized my border with India. I concentrated my forces to the west, he concentrated them to the east, and both avoided a costly stalemate in the Indian Ocean and had more forces to send elsewhere. In the future, I will seek out A for alliances more often, since I have extra reason to believe he won't betray me; this will put A in an unusually strong position.

 

This is not a unique advantage of religion; any strongly held philosophy that trumps self-interest would do. I would have made the same deal with Alicorn, who has stated loudly and publicly that she is a deontologist who has a deep personal aversion to lying2. I would have made it with Eliezer, who has a consequentialist morality but, on account of the consequences, has said he would not break an oath even for the sake of saving the world.

 

But I only trust Alicorn and Eliezer because I've discussed morality with both of them in a situation where they had no incentive to lie; it was only in the very unusual conditions of Less Wrong that they could send such a signal believably. Religion is a much easier signal to send and receive without being a moral philosopher.

 

Study Five: Excuses as Deviations from a Rule


My previous post, Eight Short Studies on Excuses, was inspired by a maneuver I pulled during a Diplomacy game.

 

I was Italy, and Turkey and I had formed a mutual alliance against Austria. As part of the alliance, we had decided not to fight over who got the lucrative neutral territories in between our empires. I would get Egypt, Turkey would get Greece and Yemen, and we would avoid the resource drain of fighting each other for them so we could both concentrate on Austria.

 

Both Turkey and I would have liked to grab the centers that had been promised to the other. But both Turkey and I knew that maintaining the general rule of alliance between us was higher utility than getting one extra territory. BUT both Turkey and I knew that the other would be loathe to break off the alliance between just because their partner had committed one little infraction. BUT both Turkey and I knew that we would have to do exactly that, or else our ally would have a carte blanche to violate whatever terms of the alliance they wanted.

 

Then India (from whom I had not yet extracted his oath) made a move towards Yemen, threatening to take it from both of us. I responded by moving a navy to Yemen, supposedly to see off the Indian menace. I then messaged Turkey, saying that although I still respected the terms of our alliance, he was clearly too weak to keep Yemen out of Indian hands, so I would be fortifying it for him, and I hoped he would have the maturity to see this as a mutually beneficial move to prevent Indian expansionism, and not get too hung up on the exact terms of our alliance.

 

The gambit worked: Turkey decided that maintaining our alliance was more important than keeping Yemen, and that because of the trouble with India my conquest of Yemen was not indicative of a general pattern of alliance-breaking that needed to be punished.

 

I can't claim total victory here: several years later, when the threat of Austria had disappeared, Turkey betrayed me and captured half my empire, partly because of my actions in Yemen.

 

Study Six: For the Sake of Revenge


This comes from the book Game Theory at Work:

 

Consider the emotion of revenge. At its core, revenge means hurting someone who has harmed you, even if you would be better off leaving him alone. Revenge is an irrational desire to harm others who have injured our loved ones or us.

 

To see the benefit of being known as vengeful, consider a small community living in prehistoric times. Imagine that a group of raiders stole food from this community. A rational community would hunt down the raiders only if the cost of doing so was not too high. A vengence-endowed community would hunt down the raiders regardless of the cost. Since the raiders would rather go after the rational community, being perceived as vengeful provides you with protection and therefore confers an evolutionary advantage.

 

I play Diplomacy often against the same people, so I decided I needed to cultivate a reputation for vengefulness. And by "decided to cultivate a reputation for vengefulness", I mean "Turkey betrayed me and I was filled with the burning rage of a thousand suns".

 

So my drive for revenge was mostly emotional instead of rational. But what I didn't do was suppress my anger, the way people are always telling you. Suppressing anger is a useful strategy for one-shot games, but in an iterated game, getting a reputation for anger is often more valuable than behaving in your immediate rational self-interest.

 

So I decided to throw the game to Germany, Turkey's biggest rival. I moved my forces away from the Italian-German border and invited Germany to take over my territory. At the same time, I used my remaining forces supporting German attacks against Turkey. The Austrians, who had been dealing with Turkey's betrayals even longer than I had, happily joined in. With our help, German forces scored several resounding victories against Turkey and pushed it back from near the top of the game down to a distant third.

 

Around the same time, Germany's other enemy France also betrayed me. So I told France I was throwing the game to Germany to punish him. No point in missing a perfectly good opportunity to cultivate a reputation for vengefulness.

 

If I had done the rational thing and excused Turkey's betrayal because it was in my self-interest to cut my losses, I could have had a mediocre end game, and Turkey's player would have happily betrayed me the next game as soon as he saw any advantage in doing so. Instead, I'm doing very poorly in the end game, but Turkey - and everyone else - will be very wary about betraying me next time around.

 

Study Seven: In-Group Bias as a Schelling Point


I made the mistake of moderating a game of Diplomacy at the SIAI House, which turned into one of the worst I've ever seen. The players were five SIAI Visiting Fellows and two of my non-SIAI friends who happened to be in the area.

 

Jasen came up with the idea of an alliance of the five SIAI players against my two friends. Although a few of the Fellows vacillated back and forth and defected a few times, he was generally able to keep the loyalty of the five Fellows until my two friends had been eliminated from the game relatively early on. Although normally the game would have continued until one of the Fellows managed to dominate the others, it was already very late and we called it a night at that point.

 

It's easy to explain what happened as an irrational in-group bias, or as "loyalty" or "patriotism" among the SIAI folk. Jasen himself explained it as a desire to prove that SIAI people were especially cooperative and especially good at game theory, which I suppose worked. But there's another, completely theoretical perspective from which to view the SIAI Alliance.

 

Imagine you are on a lifeboat with nine other people, and determine that one of the ten of you must be killed and eaten to provide sustenance to the others. You are all ready to draw lots to decide who is dinner when you shout out "Hey, instead of this whole drawing lots thing, let's kill and eat Bob!"

 

If your fellow castaways are rational agents, they might just agree. If they go with lots, each has a 10% chance of ending up dinner. If everyone just agrees on Bob, then everyone has a 0% chance of ending up dinner (except poor Bob). Nine out of ten people are better off, and nine out of ten of you vote to adopt the new plan. Whether your lifeboat decides things by majority vote or by physical violence, it doesn't look good for Bob.

 

But imagine a week later, you still haven't been rescued, and the whole situation repeats. If everyone lets you repeat your action of calling out a name, there's a 1/9 chance it'll be eir name - no better than drawing lots. In fact, since you're very unlikely to call out your own name, it's more of a 1/8 chance - worse than just drawing lots. So everyone would like to be the one who calls out the name, and as soon as the lots are taken out, everyone shouts "Hey, instead of the whole drawing lots thing, let's kill and eat X!" where X is a different person for each of the nine castaways. This is utterly useless, and you probably end up just drawing lots.

 

But suppose eight of the nine of you are blond, and one is a brunette. The brunette is now a Schelling point. If you choose to kill and eat the brunette, there's a pretty good chance all of your blond friends will do the same, even if none of you had a pre-existing prejudice against brunettes. Therefore, all eight of you shout out "Let's kill and eat the brunette!", since this is safer than drawing lots. Your lifeboat has invented in-group bias from rational principles.

 

Such alliances are equally attractive in Diplomacy. When the five SIAI Fellows allied against my two friends, they ensured there was a five-against-two alliance with themselves on the winning side, and successfully reduced the gameboard from six opponents to four. Although they could have done this with anyone (eg Jasen could have selected two other Fellows and my two friends, and forged an equivalent coalition of five), Jasen would have been at risk of five other people having the same idea and excluding him. By choosing a natural and obvious division in which he was on the majority, Jasen avoided this risk.

 

Rationalist Diplomacy


I'm interested in seeing what a Diplomacy game between Less Wrongers looks like. I'm willing to moderate. The first seven people to sign up get places (don't sign up if you don't expect to have enough time for about two or three turns/week), and the next few can be alternates. Doesn't matter if you've ever played before as long as you read the rules above and think you understand them. (We already have seven people. See the post in Discussion. If many more sign up, someone else may want to moderate a second game).

 

 

Footnotes


1: Source: "Nice Guys Finish First" in the Frameshift album Unweaving the Rainbow.

 

2. Alicorn wishes me to note that she considers anyone playing a Diplomacy game  without prior out-of-game-context agreements secured to have waived eir right to complete honesty from her, but the general principle still stands.

Diplomacy as a Game Theory Laboratory

44 Post author: Yvain 12 November 2010 10:19PM

Game theory. You've studied the posts, you've laughed at the comics, you've heard the music1. But the best way to make it Truly Part Of You is to play a genuine game, and I have yet to find any more effective than Diplomacy.

 

Diplomacy is a board game for seven people played on a map of WWI Europe. The goal is to capture as many strategic provinces ("supply centers") as possible; eighteen are needed to win. But each player's country starts off with the same sized army, and there is no luck or opportunity for especially clever tactics. The most common way to defeat an enemy is to form coalitions with other players. But your enemies will also be trying to form coalitions, and the most profitable move is often to be a "double agent", stringing both countries along as long as you can. All game moves are written in secret and revealed at the same time and there are no enforcement mechanisms, so alliances, despite their central importance, aren't always worth the paper they're printed on.

 

The conditions of Diplomacy - competition for scarce resources, rational self-interested actors, importance of coalitions, lack of external enforcement mechanisms - mirror the conditions of game theoretic situations like the Prisoner's Dilemma (and the conditions of most of human evolution!) and so make a surprisingly powerful laboratory for analyzing concepts like trust, friendship, government, and even religion.

 

Over the past few months, I've played two online games of Diplomacy. One I won through a particularly interesting method; the other I lost quite badly, but with an unusual consolation. This post is based on notes I took during the games about relevant game theoretic situations. You don't need to know the rules of Diplomacy to understand the post, but if you want a look you can find them here.

 

Study One: The Prisoner's Dilemma


The Prisoner's Dilemma is a classic case in game theory in which two players must decide whether or not to cooperate for a common goal. If both players cooperate, they both do better than if both defect, but one player can win big by defecting when the other cooperates. This situation is at the heart of Diplomacy.

 

Germany and France have agreed to ally against Britain. Both countries have demilitarized their mutual border, and are concentrating all of their forces to the north, where they take province after province of British territory.

 

But Britain is fighting back; not successfully, but every inch of territory is hard-won. France is doing well for itself and has captured a few British cities, but it could be doing better. The French player thinks to eirself: I could either continue battering against the heavily defended British lines, or I could secretly ally with Britain, stab Germany in the back, and waltz in along our undefended mutual border before the Germans even know what hit them. Instead of fighting for each inch of British land, I could be having dinner in Berlin within a week.

 

Meanwhile, in Berlin, the German player is looking towards France's temptingly undefended border and thinking the exact same thing.

 

If both France and Germany are honorable, and if both countries know the other is honorable, the two of them can continue fighting Britain with a two-to-one numerical advantage and probably divide England's lucrative territory among the two of them.

 

If Germany is naively trusting and France is a dishonest backstabber, then France can get obscene rewards by rolling over Germany while the Kaiser's armies are tied up on the fields of England.

 

If both countries are suspicious of the other, or if both countries try to backstab each other simultaneously, then they will both divert forces away from the war on England to guard their mutual border. They will not gain any territory in England, and they will not gain any territory along their border. They've not only stabbed each other in the back, they've shot themselves in the foot.

 

Study Two: Parfit's Hitch-Hiker


The wiki describes Derek Parfit's famous hitchhiker problem as:

 

Suppose you're out in the desert, running out of water, and soon to die - when someone in a motor vehicle drives up next to you. Furthermore, the driver of the motor vehicle is a perfectly selfish ideal game-theoretic agent, and even further, so are you; and what's more, the driver is Paul Ekman, who's really, really good at reading facial microexpressions. The driver says, "Well, I'll convey you to town if it's in my interest to do so - so will you give me $100 from an ATM when we reach town?"

 

Now of course you wish you could answer "Yes", but as an ideal game theorist yourself, you realize that, once you actually reachtown, you'll have no further motive to pay off the driver. "Yes," you say. "You're lying," says the driver, and drives off leaving you to die.

 

The so-called Key Lepanto opening is one of the more interesting opening strategies in Diplomacy, and one that requires guts of steel to pull off. It goes like this: Italy and Austria decide to ally against Turkey. This is common enough, and hindered by the fact that Turkey is probably expecting it and Italy's kind of far away from Turkey anyway.

 

So Italy and Austria do something unexpected. Italy swears loudly and publicly that ey's allied with Austria. Then, the first turn, Italy moves deep into undefended Austrian territory! Austria is incensed, and curses loud and long at Italy's betrayal and at eir own stupidity for leaving the frontier unguarded. Turkey laughs and leaves the two of them to their war when - boom - Austria and Italy launch a coordinated attack against Turkey from Italy's base deep in Austrian territory. The confused Turkey has no chance to organize a resistance before combined Italo-Austrian forces take Constantinople.

 

It's frequently a successful strategy, especially for Italy. You know what else is a successful strategy for Italy? Doing this up to the point where they take over lots of Austrian territory, forgetting the part where it was all just a ploy, and then ending up in control of lots of Austrian territory, after which they can fight Turkey at their leisure.

 

It's very much in Italy's advantage to play a Key Lepanto opening, and they may beg the Austrian player to go for it, saying correctly that it would benefit both of them. But the Austrian player very often refuses, telling Italy that ey would have no incentive not to just keep the conquered territory.

 

This problem resembles the Hitchhiker: Italy is the lost man, and Austria is the driver. Italy really wants Austria to help em play the awesome Key Lepanto opening, but Austria knows that ey would have no incentive not to break his promise once Austria's given him the help he needs. As a result, neither country gets what they want. The Key Lepanto opening is played only rarely, and this is one of the reasons.

 

Study Three: Enforceable Side Contracts


The Prisoner's Dilemma is nontrivial because there's no enforcement mechanism. In the presence of an enforcement mechanism, it becomes much simpler. Say two mobsters are about to be arrested, and expect to be put in a Prisoner's Dilemma type situation. They approach the mob boss with a contract with both of their names on it, saying that they have both agreed that if either of them testifies against the other, the mob boss should send his goons to shoot the rat.

 

For many payoff matrices, signing this contract will be a no-brainer. It ensures your opponent will cooperate at the relatively low cost of forcing you to cooperate yourself, and almost guarantees you safe passage into the desirable (C,C) square. Not only does it prevent your opponent doesn't defect out of sheer greed, but it prevents your opponent from worrying that you're going to defect and then defecting emself to save emself from being the chump.

 

The game of Diplomacy I won, I won through an enforceable side contract (which lost me a friend and got me some accusations of cheating, but this is par for the course for a good Diplomacy game). I was Britain; my friend H was France. H and I knew each other from an medieval times role-playing game, in which we both held land and money. The medieval kingdom of this game had a law on the books that any oath witnessed by a noble was binding on both parties and would be enforced by the king. So H and I went into our role-playing game and swore an oath before a cooperative noble, declaring that we would both aid each other in a permanent alliance in Diplomacy, or else all our in-game lands and titles would be forfeit.

 

A lot of people made fun of me for this, including H, but in my defense I did end up winning the game. H and I were able to do things that would otherwise have been impossible; for example, in order to convince our enemy Germany that we were at war, I took over the French city of Brest. Normally, this would be almost impossible for two allies to coordinate, even as a red herring, for exactly the reasons listed in the Hitchhiker problem above. Since the two of us were able to trust each other absolutely, this otherwise difficult maneuver became easy.

 

One of the advantages to strong central government is that it provides an enforcement mechanism for contracts, which benefits all parties.

 

Study Four: Religion As Enforcement


Religion is a special case of the enforceable side-contract in which God is doing the enforcing. God doesn't have to exist for this to work; as long as at least one party believes He does, the threat of punishment will be credible. The advantage of being able to easily make enforceable side contracts even in the absence of social authority may be one reason religion became so popular, and if humans do turn out to have a genetic tendency toward belief, the side contracts might have provided part of the survival advantage that spread the gene.

 

In a Youngstown Variant game (like Diplomacy, but with Eurasia instead of just Europe), I was playing Italy and after colonizing Africa was trying to juggle my forces around to defend borders with Germany, France, Turkey, and India.

 

India was played by my friend A, who I sometimes have philosophical discussions with and who I knew to be an arch-conservative religion-and-family-values type. I decided to try something which, as far as I know, no one's ever tried in a Diplomacy game before. "Do you swear in the name of God and your sacred honor that you won't attack me?" I asked.

 

"Yes," said A, and I knew he meant it, because he takes that sort of thing really seriously. I don't know if he thought he would literally go to Hell if he broke his oath, but I'm pretty sure he wasn't willing to risk it over a board game. So I demilitarized my border with India. I concentrated my forces to the west, he concentrated them to the east, and both avoided a costly stalemate in the Indian Ocean and had more forces to send elsewhere. In the future, I will seek out A for alliances more often, since I have extra reason to believe he won't betray me; this will put A in an unusually strong position.

 

This is not a unique advantage of religion; any strongly held philosophy that trumps self-interest would do. I would have made the same deal with Alicorn, who has stated loudly and publicly that she is a deontologist who has a deep personal aversion to lying2. I would have made it with Eliezer, who has a consequentialist morality but, on account of the consequences, has said he would not break an oath even for the sake of saving the world.

 

But I only trust Alicorn and Eliezer because I've discussed morality with both of them in a situation where they had no incentive to lie; it was only in the very unusual conditions of Less Wrong that they could send such a signal believably. Religion is a much easier signal to send and receive without being a moral philosopher.

 

Study Five: Excuses as Deviations from a Rule


My previous post, Eight Short Studies on Excuses, was inspired by a maneuver I pulled during a Diplomacy game.

 

I was Italy, and Turkey and I had formed a mutual alliance against Austria. As part of the alliance, we had decided not to fight over who got the lucrative neutral territories in between our empires. I would get Egypt, Turkey would get Greece and Yemen, and we would avoid the resource drain of fighting each other for them so we could both concentrate on Austria.

 

Both Turkey and I would have liked to grab the centers that had been promised to the other. But both Turkey and I knew that maintaining the general rule of alliance between us was higher utility than getting one extra territory. BUT both Turkey and I knew that the other would be loathe to break off the alliance between just because their partner had committed one little infraction. BUT both Turkey and I knew that we would have to do exactly that, or else our ally would have a carte blanche to violate whatever terms of the alliance they wanted.

 

Then India (from whom I had not yet extracted his oath) made a move towards Yemen, threatening to take it from both of us. I responded by moving a navy to Yemen, supposedly to see off the Indian menace. I then messaged Turkey, saying that although I still respected the terms of our alliance, he was clearly too weak to keep Yemen out of Indian hands, so I would be fortifying it for him, and I hoped he would have the maturity to see this as a mutually beneficial move to prevent Indian expansionism, and not get too hung up on the exact terms of our alliance.

 

The gambit worked: Turkey decided that maintaining our alliance was more important than keeping Yemen, and that because of the trouble with India my conquest of Yemen was not indicative of a general pattern of alliance-breaking that needed to be punished.

 

I can't claim total victory here: several years later, when the threat of Austria had disappeared, Turkey betrayed me and captured half my empire, partly because of my actions in Yemen.

 

Study Six: For the Sake of Revenge


This comes from the book Game Theory at Work:

 

Consider the emotion of revenge. At its core, revenge means hurting someone who has harmed you, even if you would be better off leaving him alone. Revenge is an irrational desire to harm others who have injured our loved ones or us.

 

To see the benefit of being known as vengeful, consider a small community living in prehistoric times. Imagine that a group of raiders stole food from this community. A rational community would hunt down the raiders only if the cost of doing so was not too high. A vengence-endowed community would hunt down the raiders regardless of the cost. Since the raiders would rather go after the rational community, being perceived as vengeful provides you with protection and therefore confers an evolutionary advantage.

 

I play Diplomacy often against the same people, so I decided I needed to cultivate a reputation for vengefulness. And by "decided to cultivate a reputation for vengefulness", I mean "Turkey betrayed me and I was filled with the burning rage of a thousand suns".

 

So my drive for revenge was mostly emotional instead of rational. But what I didn't do was suppress my anger, the way people are always telling you. Suppressing anger is a useful strategy for one-shot games, but in an iterated game, getting a reputation for anger is often more valuable than behaving in your immediate rational self-interest.

 

So I decided to throw the game to Germany, Turkey's biggest rival. I moved my forces away from the Italian-German border and invited Germany to take over my territory. At the same time, I used my remaining forces supporting German attacks against Turkey. The Austrians, who had been dealing with Turkey's betrayals even longer than I had, happily joined in. With our help, German forces scored several resounding victories against Turkey and pushed it back from near the top of the game down to a distant third.

 

Around the same time, Germany's other enemy France also betrayed me. So I told France I was throwing the game to Germany to punish him. No point in missing a perfectly good opportunity to cultivate a reputation for vengefulness.

 

If I had done the rational thing and excused Turkey's betrayal because it was in my self-interest to cut my losses, I could have had a mediocre end game, and Turkey's player would have happily betrayed me the next game as soon as he saw any advantage in doing so. Instead, I'm doing very poorly in the end game, but Turkey - and everyone else - will be very wary about betraying me next time around.

 

Study Seven: In-Group Bias as a Schelling Point


I made the mistake of moderating a game of Diplomacy at the SIAI House, which turned into one of the worst I've ever seen. The players were five SIAI Visiting Fellows and two of my non-SIAI friends who happened to be in the area.

 

Jasen came up with the idea of an alliance of the five SIAI players against my two friends. Although a few of the Fellows vacillated back and forth and defected a few times, he was generally able to keep the loyalty of the five Fellows until my two friends had been eliminated from the game relatively early on. Although normally the game would have continued until one of the Fellows managed to dominate the others, it was already very late and we called it a night at that point.

 

It's easy to explain what happened as an irrational in-group bias, or as "loyalty" or "patriotism" among the SIAI folk. Jasen himself explained it as a desire to prove that SIAI people were especially cooperative and especially good at game theory, which I suppose worked. But there's another, completely theoretical perspective from which to view the SIAI Alliance.

 

Imagine you are on a lifeboat with nine other people, and determine that one of the ten of you must be killed and eaten to provide sustenance to the others. You are all ready to draw lots to decide who is dinner when you shout out "Hey, instead of this whole drawing lots thing, let's kill and eat Bob!"

 

If your fellow castaways are rational agents, they might just agree. If they go with lots, each has a 10% chance of ending up dinner. If everyone just agrees on Bob, then everyone has a 0% chance of ending up dinner (except poor Bob). Nine out of ten people are better off, and nine out of ten of you vote to adopt the new plan. Whether your lifeboat decides things by majority vote or by physical violence, it doesn't look good for Bob.

 

But imagine a week later, you still haven't been rescued, and the whole situation repeats. If everyone lets you repeat your action of calling out a name, there's a 1/9 chance it'll be eir name - no better than drawing lots. In fact, since you're very unlikely to call out your own name, it's more of a 1/8 chance - worse than just drawing lots. So everyone would like to be the one who calls out the name, and as soon as the lots are taken out, everyone shouts "Hey, instead of the whole drawing lots thing, let's kill and eat X!" where X is a different person for each of the nine castaways. This is utterly useless, and you probably end up just drawing lots.

 

But suppose eight of the nine of you are blond, and one is a brunette. The brunette is now a Schelling point. If you choose to kill and eat the brunette, there's a pretty good chance all of your blond friends will do the same, even if none of you had a pre-existing prejudice against brunettes. Therefore, all eight of you shout out "Let's kill and eat the brunette!", since this is safer than drawing lots. Your lifeboat has invented in-group bias from rational principles.

 

Such alliances are equally attractive in Diplomacy. When the five SIAI Fellows allied against my two friends, they ensured there was a five-against-two alliance with themselves on the winning side, and successfully reduced the gameboard from six opponents to four. Although they could have done this with anyone (eg Jasen could have selected two other Fellows and my two friends, and forged an equivalent coalition of five), Jasen would have been at risk of five other people having the same idea and excluding him. By choosing a natural and obvious division in which he was on the majority, Jasen avoided this risk.

 

Rationalist Diplomacy


I'm interested in seeing what a Diplomacy game between Less Wrongers looks like. I'm willing to moderate. The first seven people to sign up get places (don't sign up if you don't expect to have enough time for about two or three turns/week), and the next few can be alternates. Doesn't matter if you've ever played before as long as you read the rules above and think you understand them. (We already have seven people. See the post in Discussion. If many more sign up, someone else may want to moderate a second game).

 

 

Footnotes


1: Source: "Nice Guys Finish First" in the Frameshift album Unweaving the Rainbow.

 

2. Alicorn wishes me to note that she considers anyone playing a Diplomacy game  without prior out-of-game-context agreements secured to have waived eir right to complete honesty from her, but the general principle still stands.

V is for Value Maximizing Agent: London, November 5

5 Yvain 28 October 2010 06:53PM

During the last London meetup, which I conveniently scheduled during Easter, I promised that I'd see if the next time, I could make it to London sometime that wasn't a national holiday.

The time has come to break that promise, so I will be in London for a day on Friday November 5th. If anyone wants to meet up, I'll be around that evening at 8 or so to discuss rationality-related issues, chat, or orchestrate a terrorist campaign to overthrow the government while wearing nifty masks. We can try the top floor of that same Waterstone's in Piccadilly Circus, and relocate to Starbucks if it doesn't work out. Does that work for anybody?

Comment author: NancyLebovitz 27 October 2010 11:02:03PM 1 point [-]

I've had a little more time to think-- how sure are you that the studies you cite were well-constructed?

Comment author: Yvain 27 October 2010 11:56:22PM 1 point [-]

My only evidence is that they're mostly by large and respected institutions, considered exemplary by the medical community, and that reading a one page summary of them I didn't come across anything that made me think they weren't.

Comment author: NancyLebovitz 26 October 2010 12:52:38AM 2 points [-]

You may be right about the statins-- part of what spooked me about them was running into a woman whose husband had taken permanent muscle damage from them, which suggested to me that the side effect might not be all that rare.

You mentioned that it was important for them to be correctly prescribed. How common is it for them to not be correctly prescribed?

Comment author: Yvain 26 October 2010 04:01:06PM *  10 points [-]

Rhabdomyolysis, which I think is the kind of severe permanent muscle damage you're talking about, is well-known enough as a side effect of statins that it's taught in first year medical school classes. There was one statin that may have had a relatively high (1/2,000 per year) rhabdomyolysis rate and was withdrawn from the market after a couple of years for that reason. The statins currently on the market have about a 1/20,000/year rhabdomyolysis rate, which is actually low enough that no one is entirely sure it's not background noise although no one's taking any chances. Since they also have a 1+/500/year heart attack prevention rate, they prevent something like 50 heart attacks for each case of rhabdomyolysis they cause, which seems "worth it".

Muscle damage rates increase by a lot if you take statins with fibrates (another cholesterol lowering drug). I think (not sure) that prescribing these two drugs together is Officially Discouraged, although there might be some leeway in cases of people with crazy high cholesterol. I've also heard having grapefruit juice with statins increases the risk (grapefruit juice messes with liver enzymes) but I'm not sure if that is practically important or just random clinical trivia.

As for correct prescription: I am only a student, I haven't checked the official guidelines, and if you hear otherwise from any doctor trust the doctor and not me - however, as I understand it there is pretty good evidence for giving a statin to people who have already had a cardiovascular event in order to prevent a second one, and much weaker evidence (depending on whose studies and meta-analyses you prefer) for giving it to someone who's never had a cardiovascular event. Many doctors give them to the latter category anyway just because irreversible side effects are so rare and they would rather be safe than sorry regarding heart attacks; I see some merit in both sides of the argument.

Comment author: Yvain 25 October 2010 11:38:47PM *  27 points [-]

Okay, but don't make the mistake of the guy who says "The mainstream media is all lies - so I'll only trust what I read on shady Internet conspiracy sites". Saying that there are likely flaws in mainstream medical research doesn't license you to discount any specific medical finding unless you have particular reason to believe that finding is false. And it certainly doesn't license you to place more credibility in small, poorly performed studies that contradict large, well-performed studies, or in fringe theories that contradict mainstream theories. Unless you hold your favorite theory, be it anti-vax, paleo-diet, or whatever, to the same high standard you hold the medical mainstream, every true fact you learn about flaws in medical research makes you stupider.

The study mentioned above looks at exciting cutting-edge research over the past decade. It says that 40% or so was proven wrong. This is good and to the credit of medical science! It means the system is working as it should in retesting things and getting the false stuff out. The basis of science isn't getting everything right the first time, it's making sure everyone's work gets checked and double-checked until only the truth survives. An unreplicated study in almost any area is an intriguing possibility and nothing more; medicine is no exception. If the media makes a big deal about a new study and publishes "VITAMIN B CURES BREAST CANCER!!!" in 72 point font in the newspapers, that is an interesting fact about the media and the people who believe it, but not an interesting fact about medical science.

Good doctors are both conservative and utilitarian. They stick to older, well-proven treatments unless the advantage of a new treatment is so great that it outweighs the uncertainty and risks involved. IMHO the medical consensus has been right on the important things a surprising amount of the time.

I would strongly discourage people from bewaring statins overly much. I don't see anything by Ioannidis saying the studies surrounding statins are particularly bad. Ioannidis says research is less likely to be true if it has low sample sizes, low effect sizes, bias, and a wide net. There have been several statin trials with sample sizes in the thousands to tens of thousands (see: JUPITER, SSSS, etc.) They've found that death rate from heart attacks in people correctly prescribed statin goes down by 30%, which is not at all a small effect size. Many such trials have not been linked to statin manufacturers or anyone with an axe to grind. And because people already know statins are supposed to reduce cholesterol, there is much less of a wide net than if you were to give a bunch of people statins and, say, see if any diseases became less common - the studies had a clearly designated endpoint, which they achieved.

Are there people who suggest the side effects of statins are worse than everyone else thinks? A few, and based off of very little evidence (I believe the idea that statins cause memory dysfunction is based mostly off isolated case reports, and there are only 60 out of many years of hundreds of thousands of people on statins - basically background noise). I haven't investigated this thoroughly, but the side effects would have to be pretty darned bad and pretty darned robust to stop prescribing a drug with an NNT in the two digits (ie it takes under 100 statin prescriptions to prevent one heart attack), and I treat people trying to exaggerate drug side effects as just as real a failure mode as doctors trying to exaggerate drug benefits, and use just as much caution.

The advice in the third-to-last paragraph, except perhaps the specific singling out of statins, remains excellent.

Comment author: JGWeissman 19 October 2010 12:49:23AM *  1 point [-]

When are your breaks (and when will you be home)? We are trying to do these monthly, and we could arrange the December meet up for when you are in town.

Comment author: Yvain 19 October 2010 11:11:40AM 1 point [-]

I'll let you know when I'll be home after I figure it out myself. Thanks.

Comment author: Yvain 18 October 2010 11:44:42PM 2 points [-]

Wow, that's less than five minutes from my family home. Schedule one of these during Christmas or Summer vacations and I'm in.

Re: sub-reddits

7 Yvain 17 October 2010 01:34PM

A while back, I polled the community on the possibility of subreddits. Most people said they wanted them, and I said I'd investigate.

I talked to a couple of people and eventually ended up talking to Tricycle, the developers of this site. They told me about their own proposed solution to the community organization problem, which is this new Discussion section. They said that searching the discussion section by tag was equivalent to a sub-reddit. For example, if you want a sub-reddit on consciousness, the discussion consciousness tag search is an amazing imitation

I told them I wasn't entirely convinced by this and sent some reasons why, but I haven't heard back from them lately and I'm not going keep pursuing this and make a big deal of it unless a large percentage of the people who wanted sub-reddits are unsatisfied.

Comment author: [deleted] 15 October 2010 06:18:21PM *  0 points [-]

My biggest problem when meditating is that when I focus on my breath, I switch to breathing consciously, and I can't consciously get it right.

Here's some advice from a guy named Ajahn Brahm:

A common problem at this stage is the tendency to control the breathing, and this makes the breathing uncomfortable. To overcome this problem, imagine that you are just a passenger in a car looking through the window at your breath. You are not the driver, nor a `back seat driver', so stop giving orders, let go and enjoy the ride. Let the breath do the breathing while you simply watch without interfering.

I also had this problem for a while.

In response to comment by [deleted] on A Novice Buddhist's Humble Experiences
Comment author: Yvain 15 October 2010 07:01:34PM 0 points [-]

I discovered something interesting regarding this yesterday. I mentioned that when I breathe too much, I get paraesthesias (feeling of numbness and tingling).

Well, now I've noticed that checking to see whether I have paraesthesias also causes paraesthesias. I don't know if this is true of everyone, but just thinking "I wonder if my face is tingling right now" causes my face to tingle quite perceptibly.

I think this was at the root of a lot of my worries over breathing "wrong".

Comment author: Yvain 14 October 2010 07:03:21PM *  79 points [-]

On any task more complicated than sheer physical strength, there is no such thing as inborn talent or practice effects. Any non-retarded human could easily do as well as the top performers in every field, from golf to violin to theoretical physics. All supposed "talent differential" is unconscious social signaling of one's proper social status, linked to self-esteem.

A young child sees how much respect a great violinist gets, knows she's not entitled to as much respect as that violinist, and so does badly at violin to signal cooperation with the social structure. After practicing for many years, she thinks she's signaled enough dedication to earn some more respect, and so plays the violin better.

"Child prodigies" are autistic types who don't understand the unspoken rules of society and so naively use their full powers right away. They end out as social outcasts not by coincidence but as unconscious social punishment for this defection.

Comment author: Yvain 13 October 2010 09:42:06AM 1 point [-]

Do you speak Arabic?

I've heard it said many times by Arabic-speakers that the Quran is an incredible book, unbelievably well-written and beautiful, and that its poetry is one of the factors responsible for the success of Islam.

I've also heard them say that no existing English translation is remotely as good. I would agree; I find it unbelievably boring, basically the same couple of lines about believe in Allah or you'll go to Hell over and over again (disclaimer: I've read chunks of it but not the whole thing). In terms of literary value I vastly prefer the Bible and some of the Hindu scriptures. But those have generally had better translators.

Comment author: Yvain 10 October 2010 10:32:11PM 4 points [-]

Do you have any evidence for your claim about tactic 4? I'll sometimes (as you mention) eat unhealthily when I'm unhappy, but this seems like a perfectly rational "Eating unhealthy foods makes me happy; I can't do it all the time because it would make me fat, but I can do it as an acute intervention when I'm unusually unhappy to make things a little better." This seems to work okay for me.

Comment author: Yvain 10 October 2010 02:12:15PM *  0 points [-]

To the degree that these things are really correlated, I usually interpret it as signaling being a good member of society vs. signaling being above/better than/outside of society.

But I fall pro-nice on five and anti-nice on three of your examples, which doesn't seem too far from chance. If you want to investigate this further, consider getting a few more examples, then making a poll about which sides of each issue different people are on, and seeing whether you end up with mostly half-and-halfs, or with two very different clusters of near-zeros and near-alls. Insofar as you get the two very different clusters, you'd have evidence that this is a real phenomenon worth investigating.

Comment author: Yvain 07 October 2010 10:08:02PM *  6 points [-]

This would make doctors' lives a living hell and I don't think it would be very good for patients either.

No matter how bad a drug is, there are always going to be people who get better by coincidence when they're on it. No matter how good a drug is, there are always going to be people who weren't even diagnosed with the right disease so of course the drug isn't going to cure them. No matter how safe a drug is, there's always going to be someone who coincidentally got a rash or a heart attack or something at the same time ey started the drug.

All of these people post on Internet forums, and people who have the most extreme and atypical experiences post on Internet forums the most. This is already a big problem for doctors - they prescribe Drug X, the patient goes home, looks it up on the Internet, and finds a page by some crank who had a heart attack when on Drug X and is now leading a campaign against it, and the doctor has to try to communicate that there have been big studies which show the drug has no greater heart attack rate than in the general population to people who may not be fully aware of the difference between scientific and anecdotal evidence.

Evaluating a drug on anything other than the results of randomized clinical trials, if such trials are available, is a bad idea and gets people killed. When no randomized clinical trials are available, I suppose anecdotal evidence can be useful, but only if you're an exceptionally good rationalist who can filter out the chaff.

Also, I think there's a big difference between anecdotes from people you know and anecdotes from strangers. People you know are a random sample: if you saw Uncle Jim start trying Wellbutrin, and you observed the effects, he's essentially a scientific study with a sample of one. Getting anecdotes from people on the Internet self-selects for people who want to talk about their experience.

Maybe the people who get better on a certain drug become evangelists for it all over the Internet, while people who don't get better on it just consider it one more failed drug out of dozens and never talk about it. Maybe people who get a side effect are enraged at the drug company and talk about it every chance they get while people who do fine don't talk as much. Maybe people who are depressed don't have the energy to post on health forums, so if you ask whether an antidepressant works you're only going to get the people who aren't depressed anymore, who are all going to say that it did. Maybe 100% of people on the forum who got an emergency heart attack treatment will say it works, because the rest of them are all dead.

The human tendency to trust a guy named pRoZaCuSeR331 they met on the Internet over the results of meticulously-crafted meta-analyses is an amazing thing, and I'd be wary of doing anything that might further encourage it.

The only cases in which I think this is useful are cases like erowid and nootropics, where something is so uncommon or illegal that no one's bothered to do any formal investigations on it.

Comment author: Yvain 07 October 2010 07:08:15PM *  5 points [-]

The line between genius and insanity is measured only by success

-- unknown

Comment author: Yvain 07 October 2010 07:07:18PM *  12 points [-]

A neighbor came to Nasrudin, asking to borrow his donkey. "It is out on loan," the teacher replied. At that moment, the donkey brayed loudly inside the stable. "But I can hear it bray, over there." "Whom do you believe," asked Nasrudin, "me or a donkey?"

-- old Sufi parable

Comment author: Yvain 07 October 2010 07:04:21PM 22 points [-]

"Live a good life. If there are gods and they are just, then they will not care how devout you have been, but will welcome you based on the virtues you have lived by. If there are gods, but unjust, then you should not want to worship them. If there are no gods, then you will be gone, but will have lived a noble life that will live on in the memories of your loved ones."

— Marcus Aurelius

Comment author: Yvain 07 October 2010 07:00:04PM *  32 points [-]

Even after ten thousand explanations, a fool is no wiser, but an intelligent man requires only two thousand five hundred.

-- Brahma, Mahabharata

Comment author: Yvain 04 October 2010 05:49:44PM 5 points [-]

This is really good and I'm glad you posted it and I will be trying a lot of what you mention. Also, I'm glad Google helpfully informed me that there are two different uses of "jhana", because I only knew the other one and was about to post that if you reached dhyana your first time meditating I was pretty sure you were the Maitreya Buddha.

I find I have to meditate sitting up because I become very sleepy if I do it lying down. If you can avoid sleepiness, you may escape that need. I've also heard uncomfortable positions like lotus suggested because you're trying to stop all body movement, all positions become equally uncomfortable when you can't move at all in them, and it's better to have a position you know you're supposed to be uncomfortable in so you don't try to fake a shift. That having been said, I can't stay in lotus for forty minutes.

My biggest problem when meditating is that when I focus on my breath, I switch to breathing consciously, and I can't consciously get it right. I either end up gasping for breath or hyperoxygenated (which causes paraesthesias and which I confused with sort sort of mystical body energy or something for a while until I realized what was going on). Do you not have this problem?

Are mass hallucinations a real thing?

14 Yvain 03 October 2010 08:15PM

One of the explanations in the irrationality game thread for UFOs and other paranormal events seen by multiple people at once, like the was mass hysteria. This is also a common explanation given for any seemingly paranormal event that multiple people have independently witnessed.

But mass hysteria is mostly known from incidents where people hysterically believe they have some disease, or have some hysterical delusion (false belief). In cases where people report seeing something or having a hallucination, it tends to be a few people across a large society. For example, when reports of Spring-Heeled Jack were going around England, multiple people claimed to have seen Spring-Heeled Jack, but there were no cases of hundreds of people seeing him simultaneously; therefore, the hysteria could have selected for people who were already a little bit crazy, or it could just have been that out of millions of English people a few of them were willing to say anything to get attention.

Conformity pressures can cause people to misinterpret borderline perceptions - for example, if someone says a random pattern of dots form Jesus' face, I have no trouble believing that, thus primed, people will be able to find Jesus' face in the dots. But it's a much bigger leap to assert that if I say "Jesus is standing right there in front of you" with enough conviction, you'll suddenly see him too.

Does anyone have any evidence that mass hysteria can produce a vivid hallucination shared among multiple otherwise-sane people?

Comment author: Yvain 03 October 2010 03:47:31PM 19 points [-]

I feel like I am conscious. If consciousness doesn't exist, then there must be some sort of illusion of consciousness to make me feel like it does. I see no reason not to call this illusion of consciousness "consciousness", since it fully explains what we mean when we say "consciousness".

Therefore, consciousness exists.

This sounds like some sort of horrible misuse of Dark Arts, and it would be if you plug in any other term (for example, "God"), but since "consciousness" has no properties except "a certain experience people perceive themselves as having", whatever it is that makes them perceive that experience satisfies the definition of "consciousness".

This is not an explanation of consciousness or at all helpful in the formal study of consciousness, but it's why I don't think "consciousness doesn't exist" solves anything or is particularly useful. Okay, fine, it doesn't exist, now there's still exactly the same amount of mysterious experience to explain as before.

Comment author: Will_Newsome 03 October 2010 06:30:46AM *  5 points [-]

I couldn't really understand the blog post: his theory is that there are terrestrial but nonhuman entities that like to impress the religious? But the vehicle interference cases you reference are generally not religious in nature, and are extremely varying in the actual form of the craft seen (some are red and blue, some are series of lights). What possible motivations for the entities could there be? Most agents with such advanced technology will aim to efficiently optimize for their preferences. If this is what optimizing for their preferences looks like, they have some very improbably odd preferences.

Comment author: Yvain 03 October 2010 10:38:53AM 19 points [-]

To be fair to the aliens, the actions of Westerners probably seem equally weird to Sentinel Islanders. Coming every couple of years in giant ships or helicopters to watch them from afar, and then occasionally sneaking into abandoned houses and leaving gifts?

Comment author: PlaidX 03 October 2010 05:09:52AM *  106 points [-]

Flying saucers are real. They are likely not nuts-and-bolts spacecrafts, but they are actual physical things, the product of a superior science, and under the control of unknown entities. (95%)

Please note that this comment has been upvoted because the members of lesswrong widely DISAGREE with it. See here for details.

Comment author: Yvain 03 October 2010 10:28:59AM 2 points [-]

I upvoted you because 95% is way high, but I agree with you that it's non-negligible. There's way too much weirdness in some of the cases to be easily explainable by mass hysteria or hoaxes or any of that stuff - and I'm glad you pointed out Fatima, because that was the one that got me thinking, too.

That having been said, I don't know what they are. Best guess is easter eggs in the program that's simulating the universe.

Comment author: whpearson 02 October 2010 04:46:54PM *  0 points [-]

All this AI stuff is an unnecessary distraction.

In a way yes. It was just the context that I thought of the problem under.

. Why not bomb cigarette factories? If you're willing to tell people to stop smoking, you should be willing to kill a tobacco company executive if it will reduce lung cancer by the same amount, right?

Not quite. If you are willing to donate $1000 dollars to an ad campaign against stopping smoking, because you think the ad campaign will save more than 1 life then yes it might be equivalent. If killing that executive would have a comparable effect in saving lives as the ad campaign.

Edit: To make things clearer, I mean by not donating $1000 dollars to a give well charity you are already causing someone to die.

This decision algorithm ("kill anyone whom I think needs killing") leads to general anarchy.

But we are willing to let people die who we don't think are important that we could have saved. This is equivalent to killing them, no? Or do you approach the trolley problem in some way that references the wider society?

Like I said this line of thought made me want to reject utilitarianism.

"A guy bombed a chip factory, guess we'll never pursue advanced computer technology again until we have the wisdom to use it."

That wasn't the reasoning at all! It was, "Guess the price of computer chips has gone up due to the uncertainty of building chip factories so we can only afford 6 spiffy new brain simulators this year rather than 10." Each one has an X percent chance of becoming an AGI fooming and destroying us all. It is purely a stalling for time tactic. Feel free to ignore the AI argument if you want.

Comment author: Yvain 02 October 2010 10:39:01PM 1 point [-]

I suppose the difference is whether you're doing the Intel attack now, or in a hypothetical future in which Intel is making brain simulators that seem likely to become AGI. As someone else mentioned, if we're talking about literally THEY ARE BUILDING SKYNET RIGHT NOW, then violence seems like the right idea.

I guess it is kind of suspicious that I know without doing the calculations that we're not at the point where violence is justified yet.

But we are willing to let people die who we don't think are important that we could have saved. This is equivalent to killing them, no? Or do you approach the trolley problem in some way that references the wider society?

Even though on this individual problem leaving things alone would be worse than committing an act of violence, in the general case having everyone commit acts of violence is worse than having everyone leave things alone.

This example cherry-picks a case where violence is the correct answer. But when we generalize it, most of the cases it affects won't be cherry picked and will have violence do more harm than good. We have to pretend we're setting a moral system both for ourselves and for the fundamentalist who wants to kill gay people.

So in this case, you're letting die (killing) the people your (smart) unpopular violent action would have saved, in order to save the lives of all the people whom other people's (stupid) unpopular violent actions would have killed.

It could be justified - if you're going to save the world from Skynet, that's worth instituting a moral system that gives religious fundamentalists a little more latitude to violent bigotry - but I imagine most cases wouldn't be.

Comment author: Yvain 02 October 2010 04:15:06PM 17 points [-]

All this AI stuff is an unnecessary distraction. Why not bomb cigarette factories? If you're willing to tell people to stop smoking, you should be willing to kill a tobacco company executive if it will reduce lung cancer by the same amount, right?

This decision algorithm ("kill anyone whom I think needs killing") leads to general anarchy. There are a lot of people around who believe for one reason or another that killing various people would make things better, and most of them are wrong, for example religious fundamentalists who think killing gay people will improve society.

There are three possible equilibria - the one in which everyone kills everyone else, the one in which no one kills anyone else, and the one where everyone comes together to come up with a decision procedure to decide whom to kill - ie establishes an institution with a monopoly on using force. This third one is generally better than the other two which is why we have government and why most of us are usually willing to follow its laws.

I can conceive of extreme cases where it might be worth defecting from the equilibrium because the alternative is even worse - but bombing Intel? Come on. "A guy bombed a chip factory, guess we'll never pursue advanced computer technology again until we have the wisdom to use it."

Scope insensitivity in juries

9 Yvain 02 October 2010 02:29PM

Juries found to give harsher penalties to criminals who hurt few people than to those who hurt fewer people:

http://spp.sagepub.com/content/early/2010/08/24/1948550610382308.full.pdf+html

Comment author: CannibalSmith 30 September 2010 12:26:57PM 3 points [-]

Just give us a proper forum already!

Comment author: Yvain 01 October 2010 03:13:15PM 6 points [-]

I made a poll, one of whose options was that we should have a proper forum. Not only did ZERO of the fifty-odd people choose the forum option, but it actually got down-voted in total contravention of the poll rules.

Comment author: Yvain 29 September 2010 10:39:23AM *  24 points [-]

I recently read an anecdote (so far unconfirmed) that Ataturk tried to ban the veil in Turkey, but got zero compliance from religious people, who simply ignored the law. Instead of cracking down, Ataturk decreed a second law: all prostitutes were required to wear a veil. The general custom of veil-wearing stopped immediately.

This might be the most impressive display of rationality I've ever heard of in a world leader.

Comment author: RobinZ 26 September 2010 02:06:26PM 2 points [-]

Some behavioral modifications (e.g. a program of exercise) are widely supposed to do better than nothing - would something along these lines serve as a useful placebo?

Comment author: Yvain 26 September 2010 04:31:55PM *  4 points [-]

I don't know. But placebos can have some pretty strange properties: if a doctor says a placebo is "extra strength", the patient will do better, if the pill looks colorful and complicated the patient will do better, if it's delivered via IV instead of pill the patient will do better, if the doctor wears gloves when handling the placebo because it's "so strong I can't even risk skin contact" the patient will do better, et cetera.

So there's no guarantee something flaky-sounding like "diet and exercise" would be as strong a placebo as a big name like Prozac, even if Prozac did work mostly by placebo effect, which, again, is far from certain. Try the drugs.

Comment author: ciphergoth 21 September 2010 05:03:13PM 7 points [-]

What would you do to assess the advisability of taking antidepressants?

I'm trying to advise someone who is receiving conflicting advice; there appears to be plenty of controversy in this area and many factors confound assessing the evidence. Before I point to specifics on evidence that might influence the decision one way or the other I thought I'd ask in the most general way I can. I hope using the Open Thread to play "ask a rationalist" like this isn't a bad thing!

Comment author: Yvain 26 September 2010 01:24:34PM *  0 points [-]

Whether antidepressants (realistically you're probably talking about SSRIs) do enough better than placebo in moderate cases of depression to be worthwhile is a very tough question.

But from a pragmatic viewpoint, SSRIs and placebo both do much better than nothing. Unless your friend can figure out some way to take a placebo and believe it to be effective, the SSRIs could be worth it.

Comment author: Relsqui 25 September 2010 01:06:33AM 6 points [-]

1) I don't want to alarm you, but there's a tiger next to you.

2) I second the comments about your pictures. Get someone with a real camera to take some for you, smile when they do it, and then crop/balance them.

3) You made me giggle a few times; points for that.

4) I do too message men!

5) You get to goad the reader into asking you out and still have it sound like a joke exactly once. The second time it's desperate. (You do it once in the "message me if" and once in a photo caption.)

Where in CA are you from? :)

Comment author: Yvain 25 September 2010 01:27:30AM 0 points [-]

2) I am mostly using those pictures to signal that I do interesting things; generally, the more interesting the thing I am doing, the less likely I have a good photo of it (eg when trekking in mountains I became unshaven and poorly groomed and hideous and can only take pictures by holding the camera in front of my face). I'm not really sure how to fix that. Should I get rid of all my interesting pictures?

5) Meh, good point.

I'm from Orange County, but haven't lived there in a while.

In response to comment by Larks on Politics as Charity
Comment author: JGWeissman 24 September 2010 05:33:57PM 1 point [-]

If, as in this hypothetical, Yvain is a Democrat, then he is more representative of Democrats than Republicans, and therefor is likely to acausally influence more Democrats.

Comment author: Yvain 24 September 2010 09:21:32PM 1 point [-]

I could see that as being true if my reason for not voting was "Obama just doesn't inspire me that much," but what about in the originally mentioned case where my reason is bad weather? Do you think Democrats and Republicans are different enough that their algorithms for dealing with bad weather differ in a consistent way?

Comment author: Will_Newsome 24 September 2010 05:36:57PM *  15 points [-]

OkCupid thread, anyone?

I was thinking that those of us that aren't shy could share our OkCupid profiles for critique from people who know better. (Not that we have to accept the critiques as valid, but this is an area where it'd be good to have others' opinions anyway.)

If anyone wants to get the ball rolling, post a link to your profile and hopefully someone will offer a suggestion (or a compliment).

Also, I bet cross-sexual-preference critique would be best: which for most of us means gals critiquing guys and guys critiquing gals. But I realize the LW gender skew limits that.

Comment author: Yvain 24 September 2010 09:08:18PM 8 points [-]

I don't know who erisiantaoist is, but I cannot believe he actually started his profile with the "I am slightly more committed to this group’s welfare..." quote. If I were at all gay I would date him in a second just for that.

...although I am generally surprised at how anxious people in this group are to signal transhuman weirdness, especially transhuman weirdness that only one in a few thousand people would understand or find remotely sane. Do you really have access to such a high quality dating pool that you're looking for people who will be impressed instead of confused when you name-drop AIXI and state your intention to live forever?

Comment author: Will_Newsome 24 September 2010 05:36:57PM *  15 points [-]

OkCupid thread, anyone?

I was thinking that those of us that aren't shy could share our OkCupid profiles for critique from people who know better. (Not that we have to accept the critiques as valid, but this is an area where it'd be good to have others' opinions anyway.)

If anyone wants to get the ball rolling, post a link to your profile and hopefully someone will offer a suggestion (or a compliment).

Also, I bet cross-sexual-preference critique would be best: which for most of us means gals critiquing guys and guys critiquing gals. But I realize the LW gender skew limits that.

Comment author: Yvain 24 September 2010 08:41:01PM 3 points [-]

Ah, sure. People (especially women), give me what help you can: http://www.okcupid.com/profile/ScottAlexander

Comment author: thomblake 24 September 2010 07:00:38PM 1 point [-]

I totally botched the one at the H+ summit that I had taken responsibility for, so I declared that I am no longer allowed to organize those.

Comment author: Yvain 24 September 2010 08:38:02PM *  6 points [-]

Don't feel bad. You can't possibly be worse than me. I accidentally arranged a London meet-up with two days notice on Easter in a bookstore that was closed at the time (and Tomas, Richard, and Ciphergoth showed up anyway, to their great credit).

In response to comment by Yvain on Politics as Charity
Comment author: Vladimir_Nesov 23 September 2010 08:09:10PM 1 point [-]

Do you believe that my decision to vote is as like to acausally influence my opponents into voting as it is my supporters?

For example, if the reason you were considering not voting was bad weather on election day, and you managed to discard that reason as one you won't be moved by in a voting decision, this decision would be common to many people irrespective of their candidate. By deciding to vote anyway, you establish that people in similar situations do vote.

This additionally places into question one vote as a lower estimate of influence of your decision, making it an outright useless figure.

Comment author: Yvain 23 September 2010 08:49:56PM 5 points [-]

Right, I agree with that. But let's say I'm a Democrat. If I choose to go, maybe a thousand Democrats and a thousand Republicans all choose to go, for a net gain of zero. If I choose to stay home, a thousand Democrats and a thousand Republicans choose to stay home, for a net gain of zero.

Either way, the net gain is zero. So why bother voting?

Comment author: humpolec 20 September 2010 07:23:34PM *  6 points [-]

I'm not sure if non-interference is really the best thing to precommit to - if we encounter a pre-AI civilization that still has various problems, death etc., maybe what {the AI they would have build} would have liked more is for us to help them (in a way preserving their values).

If a superintelligence discovers a concept of value-preserving help (or something like CEV?) that is likely to be universal, shouldn't it precommit to applying it to all encountered aliens?

Comment author: Yvain 23 September 2010 07:32:13PM *  0 points [-]

Voted up. It is a mortal sin to come up with responses to a valid objection I didn't anticipate to make my original theory true anyway, but for the purposes of fiction:

...perhaps the AI doesn't know whether it will be Friendly or not. The only thing the AI knows for certain is that it will further its values for it to exist. A "helpful" alien race that grants us super-technology might prevent us from accidentally creating an Unfriendly AI. Therefore, all Unfriendly AIs would regard pre-committing to helping their creator species as an acausal declaration of war.

In response to Politics as Charity
Comment author: Vladimir_Nesov 23 September 2010 05:56:55AM *  6 points [-]

She estimates that she has a 1 in 1,000,000 chance of casting the deciding vote

Why is this not a confusion? It seems on the face of it that since voters' decisions are correlated, your decision accounts for behavior of other people as well, and so you are not only casting one vote with your decision, but many votes simultaneously.

Comment author: Yvain 23 September 2010 05:54:28PM 7 points [-]

Do you believe that my decision to vote is as like to acausally influence my opponents into voting as it is my supporters? If so, and if we can expect about equal amounts of both, doesn't that produce the same problem?

In response to Politics as Charity
Comment author: b1shop 23 September 2010 02:51:08PM 10 points [-]

I can think of one big advantage of charity over politics: Giving to politics is rent-seeking behavior. If you give money to support politicians against/for dairy tariffs, then you encourage the other side to give money to their politicians who are for/against dairy tariffs. In this game matrix, "cooperate" is don't give money.

I have a hard time swallowing the ten trillion dollar figure. I don't think the two parties are really that different, and there's good, game theory for them to be that way.

In response to comment by b1shop on Politics as Charity
Comment author: Yvain 23 September 2010 05:51:51PM 28 points [-]

I think there is a market for some sort of organization by which a person who wants to donate $50 to the Democrats can contact a person who wants to donate $50 to the Republicans and mutually agree to donate their collective $100 to some third-party cause like world hunger instead.

Comment author: mattnewport 22 September 2010 11:37:51PM 3 points [-]

Gamblers are maximizing expected utility, not expected cash. That is all.

Comment author: Yvain 23 September 2010 05:43:38PM 10 points [-]

It's not all. Pramipexole and other dopamine agonist medications can cause compulsive gambling in previous non-gamblers as a side effect. That makes me think that the thrill of gambling has something to do with the dopamine system and the design of the human risk/reward system, and that compulsive gambling probably has some kind of organic cause that you couldn't find in the pure mathematics of expected utility.

Comment author: jacob_cannell 22 September 2010 06:58:46PM 3 points [-]

Some initial questions, before I even read into any of the controversy:

  • Would there be an expected evolutionary benefit for Ad-36 to increase obesity?
  • How can we distinguish correlation and causation in this case? One possible confounding variable - high vitamin D levels are positively correlated with viral resistance and negatively correlated with body fat.
  • Have we or can we try this on experimental animal models? A controlled labortory test is the most effective tool for testing the biological effects of isolated factors (as in koch's postulates)
Comment author: Yvain 22 September 2010 08:05:17PM *  16 points [-]

Would there be an expected evolutionary benefit for Ad-36 to increase obesity?

I wouldn't worry too much about this. A lot of pathogen effects have nothing to do with evolution directly, just with the human body being very complicated and it being very hard to do one thing without doing twenty other things at the same time.

For example, some scientists think a Coxsackie virus indirectly causes diabetes. The virus infects, the immune system does its usual job of mobilizing to fight the virus, but something about the virus "looks" enough like beta cells in the pancreas that the immune system gets confused and destroys them too. This doesn't provide any benefit to the virus afaik, it's just something that happens when you have a finite number of ways biological molecules can be configured.

Likewise, chlamydia can make people infertile, not because there's some evolutionary benefit to chlamydia in making people infertile, but just because fertility's a complicated process and in the process of invading the uterus chlamydia ruins all those fine-tuned organs.

I don't know anything about this particular virus, but if it does increase obesity I would expect it to be more like the Coxsackie virus and less like some weird evolutionary adaptation (although those aren't unheard of - see eg toxoplasma)

Comment author: gwern 22 September 2010 03:07:26PM 0 points [-]

So what did you find out?

Comment author: Yvain 22 September 2010 07:54:47PM 1 point [-]

That it is very difficult to learn what supplements are and aren't legal in Ireland, especially since the appropriate government website seems permanently down. If anyone has a good idea how you'd go about it, let me know. Otherwise, I'm going to wait until I see my pharmacist friend and ask her how you find these things out.

Comment author: Yvain 19 September 2010 12:51:05AM *  8 points [-]

A piece of Singularity-related fiction with a theory for your evaluation: The Demiurge's Elder Brother

Comment author: Yvain 14 September 2010 08:40:53PM *  49 points [-]

Upvote this if, out of the solution set [keep things they way they are, have subreddits, have bulletin board], you would prefer to have subreddits.

Comment author: Yvain 16 September 2010 09:45:02PM *  2 points [-]

This won the poll, so I'm going to talk to some people and see how it would get done and what it would take. I'll report back when I get answers, maybe in an Open Thread or somewhere.

Comment author: wedrifid 16 September 2010 09:19:59AM 2 points [-]

So, nootropics experts, can you think of a drug or supplement that needs testing, that you think will return measurable results in a study of only 10-20 people, that works quickly (ie no "have to take this for a month before seeing effects", preferably <2 hr onset of action) and which is sold over-the-counter in Europe?

Sulbutiamine could use some more study and is significant enough in effects that it would show a difference on that scale of investigation.

Comment author: Yvain 16 September 2010 07:42:46PM 2 points [-]

Thanks, that's my first choice right now. Just got to find out how to find out if it's legal here.

Comment author: Yvain 15 September 2010 02:49:19PM 44 points [-]

I also recently noticed this triad:

Seek sex + money / pursue only pure truth and virtue / seek sex + money

Comment author: XFrequentist 15 September 2010 12:03:15AM *  9 points [-]

Even though most people liked the idea in that recent PUA thread, I don't see any concrete moves in that direction either.

Seriously? That's a pretty quick judgement! I wrote most of a follow-up post, but I'm going to reevaluate it a bit in light of Patri's article.

I strongly support proposal 1, and I'd welcome some monitoring to make sure I don't violate this new norm.

If the subreddits idea wins, I will also chip in for the technical cost. Social.lesswrong.com seems like a decent way to do the thing-that-isn't-PUA.

Comment author: Yvain 15 September 2010 02:46:23PM 3 points [-]

You're right, I was rounding you to the nearest cliche of the last few people who said this sort of thing, and I was wrong.

Comment author: patrissimo 15 September 2010 05:19:33AM *  2 points [-]
  1. Didn't Sarah C just have a big post about this as a fallacy?

Most of these threads have been phrased along the lines of "Someone really ought to do something about this", and then everyone agrees that yeah, they should, and then nothing ever comes out of it. That's a natural phenomenon in...

I think it's a natural phenomenon on a blog - a format which is so anti-growth, so focused on shininess, that even energy towards productive change, when directed through the blog, goes nowhere. One big reason is the community norm of this all being free stuff done in spare time (except for Eliezer). Helping people grow, and designing curricula for and monitoring their growth, is hard work. It requires professional time and getting paid.

I do X all the time in my life and in my organization. The question is whether someone will take the time to create X for others. I am happy to participate in figuring out how to do X by supplying some of my very limited time. I will pay for X (workshops, coaching, or instruction), if X is taught more effectively from this community than from the many other places offering to help me grow and become better at achieving my goals. That demand will create it's own supply.

Re: subreddits & bulletin boards - Great, more shiny ways to waste people's time. Real change happens from what you do off the internet, is that so hard an idea to understand?

Comment author: Yvain 15 September 2010 02:45:39PM 10 points [-]
  1. I have re-read the Affect Heuristic post, and I don't see its relevance. Explain?

One of the last posts on this sort of thing mentioned the phrase "'Good enough' is the enemy of 'at all'".

Yes, the best way to do this would have in-person groups with paid instructors. I interpreted you as saying we should go create these groups. If your point was that these groups already exist and we should get off Less Wrong and go to them, then I misunderstood, but I am still doubtful. The vast majority of people don't have access to them (live in smaller cities without such groups, don't have time for such groups, et cetera), those who do probably don't know it, and among those who do have access and know it but still haven't joined, saying "You ought to be going to these!" is unlikely to change many minds.

But I understood you to mean that Less Wrong should work to create such groups. If that's true, then they're unlikely to happen. Only a tiny handful of cities have enough Less Wrongers to form a group, and as far as I know only the Bay Area and NYC (possibly also Southern CA?) actually have one that meets consistently and with defined agendas. That immediately excludes all LWers who live outside Bay Area and NYC. For example, I live 150 miles from the nearest other LWer.

So if we want to use the community for this, we need some way of number one organizing existing and potential big-city-groups better, and number two creating online groups for people not in big cities. If someone is good enough to be worth money, we need a way to organize and fundraise for them.

Finally, a realspace organization with a paid instructor represents a big commitment for the members and a huge commitment for the instructor. It may be that just mentioning the possibility will convince a few people to set up the necessary organization. But I think it's much more likely that people will do this after the organization has already existed in online form for a while and proven it has potential; ie using an online form to bootstrap a realspace form.

Comment author: Yvain 15 September 2010 02:23:39PM 8 points [-]

Next year I have to do a study as a school project. I will probably have access to about 10-20 test subjects, maybe more if I beg, and a lab with a full battery of professional-level cognitive tests. I'd like to study nootropic drugs and see how well they work, but the only one I'm really familiar with, piracetam, is by prescription only in Ireland, and I'm not likely to be able to prescribe it for this study.

So, nootropics experts, can you think of a drug or supplement that needs testing, that you think will return measurable results in a study of only 10-20 people, that works quickly (ie no "have to take this for a month before seeing effects", preferably <2 hr onset of action) and which is sold over-the-counter in Europe?

Comment author: Yvain 14 September 2010 08:39:04PM *  37 points [-]

To get more meta, not only has Less Wrong not produced "results", but all the posts saying Less Wrong needs to produce more "results" (example: Instrumental Rationality Is A Chimera) haven't produced any results. Even though most people liked the idea in that recent PUA thread, I don't see any concrete moves in that direction either.

Most of these threads have been phrased along the lines of "Someone really ought to do something about this", and then everyone agrees that yeah, they should, and then nothing ever comes out of it. That's a natural phenomenon in an anarchy where no one is the Official Doer of Difficult Things That Need To Be Done. Our community has one leader, Eliezer, and he has much better things to do with his time. Absent a formal organization, no one is going to be able to move a few hundred people to do things differently.

But small interventions can have major changes on behavior (see the sentence beginning with "I was reminded of this recently..." here). For example, I think if there were socialskills.lesswrong.com and health.lesswrong.com subcommunities linked to the top of the page, they would auto-populate with a community and interesting posts. I would love to see a discussion forum on nootropics where people can post their experiences and questions in an organized and easy to find way, for example. This idea has been brought up since forever and no one has ever done anything about it. The alternate idea, that we make a bulletin board in which these things can be done easily and naturally (AND WHICH CAN HANDLE OPEN THREADS IN A SANE WAY) has also been brought up since forever and no one has done anything about it (one person made a bulletin board back in the Overcoming Bias days, but no one used it. Go figure.)

So I propose the following:

  1. Community norm against saying "It would be nice if someone in our community did X" if you have no particular plans to do X and no reason to think anyone else will.

  2. Poll on whether people want a bulletin board or subreddits. This poll is below this comment.

  3. If people want a bulletin board, and they promise to actually use it once it is made, and Eliezer and Tricycle don't want to make it themselves, and no one else more competent with computers will make it, I will make and host it (maybe. I'm not sure how much traffic it would get and I don't want to commit to something that would bankrupt me. But in principle, yes.)

  4. I don't know how to program subreddits, but if that solution wins the poll, I will pay someone who does know a small amount of money to do it, and other people probably will too (because we will do the fundraising in a rationalist way!) adding up to a medium amount of money.

Comment author: Yvain 14 September 2010 08:41:24PM 10 points [-]

Upvote this if, out of the solution set [keep things they way they are, have subreddits, have bulletin board], you like the way things are now.

Comment author: Yvain 14 September 2010 08:39:04PM *  37 points [-]

To get more meta, not only has Less Wrong not produced "results", but all the posts saying Less Wrong needs to produce more "results" (example: Instrumental Rationality Is A Chimera) haven't produced any results. Even though most people liked the idea in that recent PUA thread, I don't see any concrete moves in that direction either.

Most of these threads have been phrased along the lines of "Someone really ought to do something about this", and then everyone agrees that yeah, they should, and then nothing ever comes out of it. That's a natural phenomenon in an anarchy where no one is the Official Doer of Difficult Things That Need To Be Done. Our community has one leader, Eliezer, and he has much better things to do with his time. Absent a formal organization, no one is going to be able to move a few hundred people to do things differently.

But small interventions can have major changes on behavior (see the sentence beginning with "I was reminded of this recently..." here). For example, I think if there were socialskills.lesswrong.com and health.lesswrong.com subcommunities linked to the top of the page, they would auto-populate with a community and interesting posts. I would love to see a discussion forum on nootropics where people can post their experiences and questions in an organized and easy to find way, for example. This idea has been brought up since forever and no one has ever done anything about it. The alternate idea, that we make a bulletin board in which these things can be done easily and naturally (AND WHICH CAN HANDLE OPEN THREADS IN A SANE WAY) has also been brought up since forever and no one has done anything about it (one person made a bulletin board back in the Overcoming Bias days, but no one used it. Go figure.)

So I propose the following:

  1. Community norm against saying "It would be nice if someone in our community did X" if you have no particular plans to do X and no reason to think anyone else will.

  2. Poll on whether people want a bulletin board or subreddits. This poll is below this comment.

  3. If people want a bulletin board, and they promise to actually use it once it is made, and Eliezer and Tricycle don't want to make it themselves, and no one else more competent with computers will make it, I will make and host it (maybe. I'm not sure how much traffic it would get and I don't want to commit to something that would bankrupt me. But in principle, yes.)

  4. I don't know how to program subreddits, but if that solution wins the poll, I will pay someone who does know a small amount of money to do it, and other people probably will too (because we will do the fundraising in a rationalist way!) adding up to a medium amount of money.

Comment author: Yvain 14 September 2010 08:40:53PM *  49 points [-]

Upvote this if, out of the solution set [keep things they way they are, have subreddits, have bulletin board], you would prefer to have subreddits.

Comment author: Yvain 14 September 2010 08:39:04PM *  37 points [-]

To get more meta, not only has Less Wrong not produced "results", but all the posts saying Less Wrong needs to produce more "results" (example: Instrumental Rationality Is A Chimera) haven't produced any results. Even though most people liked the idea in that recent PUA thread, I don't see any concrete moves in that direction either.

Most of these threads have been phrased along the lines of "Someone really ought to do something about this", and then everyone agrees that yeah, they should, and then nothing ever comes out of it. That's a natural phenomenon in an anarchy where no one is the Official Doer of Difficult Things That Need To Be Done. Our community has one leader, Eliezer, and he has much better things to do with his time. Absent a formal organization, no one is going to be able to move a few hundred people to do things differently.

But small interventions can have major changes on behavior (see the sentence beginning with "I was reminded of this recently..." here). For example, I think if there were socialskills.lesswrong.com and health.lesswrong.com subcommunities linked to the top of the page, they would auto-populate with a community and interesting posts. I would love to see a discussion forum on nootropics where people can post their experiences and questions in an organized and easy to find way, for example. This idea has been brought up since forever and no one has ever done anything about it. The alternate idea, that we make a bulletin board in which these things can be done easily and naturally (AND WHICH CAN HANDLE OPEN THREADS IN A SANE WAY) has also been brought up since forever and no one has done anything about it (one person made a bulletin board back in the Overcoming Bias days, but no one used it. Go figure.)

So I propose the following:

  1. Community norm against saying "It would be nice if someone in our community did X" if you have no particular plans to do X and no reason to think anyone else will.

  2. Poll on whether people want a bulletin board or subreddits. This poll is below this comment.

  3. If people want a bulletin board, and they promise to actually use it once it is made, and Eliezer and Tricycle don't want to make it themselves, and no one else more competent with computers will make it, I will make and host it (maybe. I'm not sure how much traffic it would get and I don't want to commit to something that would bankrupt me. But in principle, yes.)

  4. I don't know how to program subreddits, but if that solution wins the poll, I will pay someone who does know a small amount of money to do it, and other people probably will too (because we will do the fundraising in a rationalist way!) adding up to a medium amount of money.

Comment author: Yvain 14 September 2010 08:40:17PM 0 points [-]

Upvote this if, out of the solution set [keep things they way they are, have subreddits, have bulletin board], you prefer to have a bulletin board, and you would use it and check it often if it existed

Comment author: Yvain 14 September 2010 08:39:04PM *  37 points [-]

To get more meta, not only has Less Wrong not produced "results", but all the posts saying Less Wrong needs to produce more "results" (example: Instrumental Rationality Is A Chimera) haven't produced any results. Even though most people liked the idea in that recent PUA thread, I don't see any concrete moves in that direction either.

Most of these threads have been phrased along the lines of "Someone really ought to do something about this", and then everyone agrees that yeah, they should, and then nothing ever comes out of it. That's a natural phenomenon in an anarchy where no one is the Official Doer of Difficult Things That Need To Be Done. Our community has one leader, Eliezer, and he has much better things to do with his time. Absent a formal organization, no one is going to be able to move a few hundred people to do things differently.

But small interventions can have major changes on behavior (see the sentence beginning with "I was reminded of this recently..." here). For example, I think if there were socialskills.lesswrong.com and health.lesswrong.com subcommunities linked to the top of the page, they would auto-populate with a community and interesting posts. I would love to see a discussion forum on nootropics where people can post their experiences and questions in an organized and easy to find way, for example. This idea has been brought up since forever and no one has ever done anything about it. The alternate idea, that we make a bulletin board in which these things can be done easily and naturally (AND WHICH CAN HANDLE OPEN THREADS IN A SANE WAY) has also been brought up since forever and no one has done anything about it (one person made a bulletin board back in the Overcoming Bias days, but no one used it. Go figure.)

So I propose the following:

  1. Community norm against saying "It would be nice if someone in our community did X" if you have no particular plans to do X and no reason to think anyone else will.

  2. Poll on whether people want a bulletin board or subreddits. This poll is below this comment.

  3. If people want a bulletin board, and they promise to actually use it once it is made, and Eliezer and Tricycle don't want to make it themselves, and no one else more competent with computers will make it, I will make and host it (maybe. I'm not sure how much traffic it would get and I don't want to commit to something that would bankrupt me. But in principle, yes.)

  4. I don't know how to program subreddits, but if that solution wins the poll, I will pay someone who does know a small amount of money to do it, and other people probably will too (because we will do the fundraising in a rationalist way!) adding up to a medium amount of money.

Comment author: owevr 14 September 2010 03:47:47PM 2 points [-]

Great article. However, why do you call them "meta-contrarian", instead of "anti-contrarian"? I would not call something "meta-" unless it adds additional dimensions to the given context. For example, "meta-theory" is not about disputing particular theories but something totally different.

Comment author: Yvain 14 September 2010 07:49:03PM 1 point [-]

I interpret meta- to mean "one level above"; thus for example Douglas Hofstadter's "meta-agnostic", someone who is agnostic about agnosticism, and your own mention of "meta-theory", a theory about a theory.

I use "meta-contrary" because it's a position deliberately taken to be contrary to a position deliberately taken to be contrary.

Comment author: Apprentice 14 September 2010 03:36:29PM 5 points [-]

The pleasure I get out of trolling atheists definitely has a meta-contrarian component to it. When I was a teenager I would troll Christians but I've long since stopped finding that even slightly challenging or fun.

Comment author: Yvain 14 September 2010 07:43:12PM 7 points [-]

Yes, I often find myself tempted to do that too. Although I understand on an intellectual level that creationism is stupid, it is hard for me to get worked up about it and I certainly don't have the energy to argue with creationists ad nauseum. I do find myself angry whenever an atheist makes a superficial or stupid point in defense of atheism, or when they get too smug about how much smarter they are than creationists.

My guess is that I have a sufficiently inflated view of my intelligence to be high enough that I have no need to differentiate myself intellectually from creationists, but I do feel a need to differentiate myself intellectually from the less intelligent sort of atheist.

Comment author: Mercy 14 September 2010 12:50:58PM 3 points [-]

I'm a little confused, what purpose does this distinction serve? That people like to define their opinions as a rebellion against received opinion isn't novel. What you seem to be saying is: defining yourself against an opinion which is seen as contrarian sends a reliably different social signal to defining yourself against an opinion which is mainstream, is that a fair assessment? Because this only works if there is a singular, visible mainstream, which is obviously available in fashion but rare in the realm of ideas.

Moreover, if order-of-contrariness doesn't convey information, I can't see any situation in which one it would be helpful to indicate a positions order, where it wouldn't be just as easy and far more informative to point out the specific chain of it's controversy.

In any case I take some issue with a bunch of your example.

Firstly on feminism the obvious mainstream controversy/metacontroversy dynamic for misogyny is between second and third wave feminism in academia, and between "all sex is rape" and "pole dancing is empowering/Madonna is a feminist icon" in the media. Picking an obscure internet phenomenon closer to the starting point is blatant cherry picking.

Similarly the Bad Samaritans/New Development argument has a lot more currency than the aid is the problem one, but again that's further from both positions. For that matter the same applies to liberterianism and it's real Laius, socialism.

The number of global warming skeptics who jumped straight from "it's not happening" to "well we didn't do it" to "well we can't do anything about it without doing more harm than good" should also, combined with the overlap in arguments between self identified MRAs and younger misogynists of the "straight white christian men are the most oppressed minority" variety, give us a bit of pause. If there's any use to identifying meta contrarian positions, it has to be in distinguishing between genuine attempts to correct falsehoods made in overeager argument with the old mainstream, and sophisticated apologetics for previously exploded positions.

On second thought, convincing as I find the Stern report, enough economists argued against reducing carbon emissions on cost-benefit grounds from the beginning that the meta position deserves honest consideration. I'd like to propose instead deism as the canonical example for bad faith apologia in meta-contrarianist drag, and third wave feminism for the honest position. Is this suitably uncontroversial?

Comment author: Yvain 14 September 2010 07:38:28PM 2 points [-]

I'm a little confused, what purpose does this distinction serve? That people like to define their opinions as a rebellion against received opinion isn't novel.

You're right, the examples were pretty cherry-picked.

My point was to show that, although we tend to celebrate our failure to be lured into holding contrarian positions for the sake of contrarianism, this can itself be a trap that we need to watch out for. I think the idea of meta-contrarian-ness is novel in a way the idea of contrarian-ness is not.

Comment author: knb 14 September 2010 06:12:59AM *  16 points [-]

conservative / liberal / libertarian

No way, I don't buy this one at all. I find that most little kids are essentially naive liberals. We should give poor sick people free medicine! We should stop bad polluters from hurting birds and trees! Conservatism/libertarianism is the contrarian position. Everything has a cost! There are no free lunches! Managerial-technocratic liberals are the meta-contrarians. So what about the costs? We've got 800 of the smartest guys from Yarvard and Oxbridge to do cost-benefit analyses for us!

Of course there are meta-meta-contrarians as well: reactionaries, meta-libertarians (Patri Friedman is a good example of a metalibertarian IMO), anarchists, etc.

It's contrarians all the way down.

Comment author: Yvain 14 September 2010 07:34:15PM *  13 points [-]

I was thinking more in terms of conservative values like "My country is the best" and "Our enemies are bad people who hate our freedom", but your way makes a lot of sense too.

Although it's worth noting that all of what you say is obvious even to little kids are things no one had even thought of a hundred years ago. Rachel Carson and Silent Spring are remembered as iconic because they kick-started an environmentalist movement that just didn't really exist before the second half of the 20th century (although Thoureau and people like that get honorable mention). The idea of rich people paying to give poor sick people free medicine would have gotten you laughed out of most socially stratified civilizations on the wrong side of about 1850.

But I don't want to get too bogged down in which side is more contrarian, because it sounds too close to arguing whether liberalism or conservativism is better, which of course would be a terribly low status thing to do on a site like this :)

I think it was probably a mistake to include such large-scale politics on there at all. Whether a political position seems natural or contrarian depends on what social context someone's in, what age they are, and what the particular issue involved is.

What about this: moderately smart teenagers become extreme liberals to be contrary to the conservative ideals of their elders; excessively smart teenagers become extreme libertarians to be contrary to moderately smart teenagers and their elders, and older people become conservative (or moderate liberals) to signal they're not teenagers :)

Comment author: ata 14 September 2010 12:41:31AM *  2 points [-]

Thus Eliezer's title for this mentality, "Pretending To Be Wise".

Have we broadened that term to refer to... well, lowercase pretending-to-be-wise in general? In the original post, he used it specifically to refer to those who try to signal wisdom by neutrality. (Though I did notice he used it in the broader sense in HPMoR. Is it thus officially redefined?)

Comment author: Yvain 14 September 2010 07:14:31PM 0 points [-]

Yeah, I was thinking of the HPatMOR usage. It's a good phrase, and it would be a shame not to use it.

Comment author: HughRistik 13 September 2010 11:47:41PM 2 points [-]

Are there also meta-meta-contrarians?

Comment author: Yvain 14 September 2010 07:12:49PM 4 points [-]

Maybe it's context dependent. If I am hanging around a lot of contrarians, I usually end up looking for a meta-contrarian position. If I'm hanging around a lot of meta-contrarians who I think aren't as smart as me, and those meta-contrarians are being really smug and annoying, I become meta-meta-contrarian. I fondly remember a period of my life when I went to my college's Objectivist club every week to argue vehemently against everything they said. I think that qualifies as meta-meta-contrarian if anything does.

Comment author: steven0461 13 September 2010 11:37:25PM 14 points [-]

Global warming was the least controversial example you could think of? Seriously?

Comment author: Yvain 14 September 2010 07:10:24PM 23 points [-]

Well, the example was to show that there are certain meta-contrarian views held by a big part of this community which are trivially wrong and proof that they have gone too far. Given that restriction, what less controversial example would you have preferred?

I really would have liked to use the racism example, because it's most elegant. The in-group bias means people will naturally like their own race more than others. Some very intelligent and moral people come up with the opposing position that all races are equal; overcoming one's biases enough to believe this becomes (rightly) correlated with high intelligence and morality. This contrarian idea spreads until practically everyone believes it and signals it so much as to become annoying and inane. This creates a niche for certain people to signal their difference to the majority by becoming pro-racial differences. But taken too far, this meta-contrarian position could easily lead to racism.

But any post that includes a whole paragraph on racism automatically ends up with the comments entirely devoted to discussing racism, and the rest of the post completely ignored. Feminism would also have worked, but I would have to be dumb as a rock to voluntarily bring up gender issues on this blog. Global warming seemed like something that Less Wrong is generally willing to admit is correct and doesn't care that much about, while still having enough of an anti-global-warming faction to work as an example.

Intellectual Hipsters and Meta-Contrarianism

147 Post author: Yvain 13 September 2010 09:36PM

Related to: Why Real Men Wear Pink, That Other Kind of Status, Pretending to be Wise, The "Outside The Box" Box

WARNING: Beware of things that are fun to argue -- Eliezer Yudkowsky

Science has inexplicably failed to come up with a precise definition of "hipster", but from my limited understanding a hipster is a person who deliberately uses unpopular, obsolete, or obscure styles and preferences in an attempt to be "cooler" than the mainstream. But why would being deliberately uncool be cooler than being cool?

As previously discussed, in certain situations refusing to signal can be a sign of high status. Thorstein Veblen invented the term "conspicuous consumption" to refer to the showy spending habits of the nouveau riche, who unlike the established money of his day took great pains to signal their wealth by buying fast cars, expensive clothes, and shiny jewelery. Why was such flashiness common among new money but not old? Because the old money was so secure in their position that it never even occurred to them that they might be confused with poor people, whereas new money, with their lack of aristocratic breeding, worried they might be mistaken for poor people if they didn't make it blatantly obvious that they had expensive things.

The old money might have started off not buying flashy things for pragmatic reasons - they didn't need to, so why waste the money? But if F. Scott Fitzgerald is to be believed, the old money actively cultivated an air of superiority to the nouveau riche and their conspicuous consumption; not buying flashy objects becomes a matter of principle. This makes sense: the nouveau riche need to differentiate themselves from the poor, but the old money need to differentiate themselves from the nouveau riche.

This process is called countersignaling, and one can find its telltale patterns in many walks of life. Those who study human romantic attraction warn men not to "come on too strong", and this has similarities to the nouveau riche example. A total loser might come up to a woman without a hint of romance, promise her nothing, and demand sex. A more sophisticated man might buy roses for a woman, write her love poetry, hover on her every wish, et cetera; this signifies that he is not a total loser. But the most desirable men may deliberately avoid doing nice things for women in an attempt to signal they are so high status that they don't need to. The average man tries to differentiate himself from the total loser by being nice; the extremely attractive man tries to differentiate himself from the average man by not being especially nice.

In all three examples, people at the top of the pyramid end up displaying characteristics similar to those at the bottom. Hipsters deliberately wear the same clothes uncool people wear. Families with old money don't wear much more jewelry than the middle class. And very attractive men approach women with the same lack of subtlety a total loser would use.1

If politics, philosophy, and religion are really about signaling, we should expect to find countersignaling there as well.


Pretending To Be Wise

Let's go back to Less Wrong's long-running discussion on death. Ask any five year old child, and ey can tell you that death is bad. Death is bad because it kills you. There is nothing subtle about it, and there does not need to be. Death universally seems bad to pretty much everyone on first analysis, and what it seems, it is.

But as has been pointed out, along with the gigantic cost, death does have a few small benefits. It lowers overpopulation, it allows the new generation to develop free from interference by their elders, it provides motivation to get things done quickly. Precisely because these benefits are so much smaller than the cost, they are hard to notice. It takes a particularly subtle and clever mind to think them up. Any idiot can tell you why death is bad, but it takes a very particular sort of idiot to believe that death might be good.

So pointing out this contrarian position, that death has some benefits, is potentially a signal of high intelligence. It is not a very reliable signal, because once the first person brings it up everyone can just copy it, but it is a cheap signal. And to the sort of person who might not be clever enough to come up with the benefits of death themselves, and only notices that wise people seem to mention death can have benefits, it might seem super extra wise to say death has lots and lots of great benefits, and is really quite a good thing, and if other people should protest that death is bad, well, that's an opinion a five year old child could come up with, and so clearly that person is no smarter than a five year old child. Thus Eliezer's title for this mentality, "Pretending To Be Wise".

If dwelling on the benefits of a great evil is not your thing, you can also pretend to be wise by dwelling on the costs of a great good. All things considered, modern industrial civilization - with its advanced technology, its high standard of living, and its lack of typhoid fever -  is pretty neat. But modern industrial civilization also has many costs: alienation from nature, strains on the traditional family, the anonymity of big city life, pollution and overcrowding. These are real costs, and they are certainly worth taking seriously; nevertheless, the crowds of emigrants trying to get from the Third World to the First, and the lack of any crowd in the opposite direction, suggest the benefits outweigh the costs. But in my estimation - and speak up if you disagree - people spend a lot more time dwelling on the negatives than on the positives, and most people I meet coming back from a Third World country have to talk about how much more authentic their way of life is and how much we could learn from them. This sort of talk sounds Wise, whereas talk about how nice it is to have buses that don't break down every half mile sounds trivial and selfish..

So my hypothesis is that if a certain side of an issue has very obvious points in support of it, and the other side of an issue relies on much more subtle points that the average person might not be expected to grasp, then adopting the second side of the issue will become a signal for intelligence, even if that side of the argument is wrong.

This only works in issues which are so muddled to begin with that there is no fact of the matter, or where the fact of the matter is difficult to tease out: so no one tries to signal intelligence by saying that 1+1 equals 3 (although it would not surprise me to find a philosopher who says truth is relative and this equation is a legitimate form of discourse).

Meta-Contrarians Are Intellectual Hipsters

A person who is somewhat upper-class will conspicuously signal eir wealth by buying difficult-to-obtain goods. A person who is very upper-class will conspicuously signal that ey feels no need to conspicuously signal eir wealth, by deliberately not buying difficult-to-obtain goods.

A person who is somewhat intelligent will conspicuously signal eir intelligence by holding difficult-to-understand opinions. A person who is very intelligent will conspicuously signal that ey feels no need to conspicuously signal eir intelligence, by deliberately not holding difficult-to-understand opinions.

According to the survey, the average IQ on this site is around 1452. People on this site differ from the mainstream in that they are more willing to say death is bad, more willing to say that science, capitalism, and the like are good, and less willing to say that there's some deep philosophical sense in which 1+1 = 3. That suggests people around that level of intelligence have reached the point where they no longer feel it necessary to differentiate themselves from the sort of people who aren't smart enough to understand that there might be side benefits to death. Instead, they are at the level where they want to differentiate themselves from the somewhat smarter people who think the side benefits to death are great. They are, basically, meta-contrarians, who counter-signal by holding opinions contrary to those of the contrarians' signals. And in the case of death, this cannot but be a good thing.

But just as contrarians risk becoming too contrary, moving from "actually, death has a few side benefits" to "DEATH IS GREAT!", meta-contrarians are at risk of becoming too meta-contrary.

All the possible examples here are controversial, so I will just take the least controversial one I can think of and beg forgiveness. A naive person might think that industrial production is an absolute good thing. Someone smarter than that naive person might realize that global warming is a strong negative to industrial production and desperately needs to be stopped. Someone even smarter than that, to differentiate emself from the second person, might decide global warming wasn't such a big deal after all, or doesn't exist, or isn't man-made.

In this case, the contrarian position happened to be right (well, maybe), and the third person's meta-contrariness took em further from the truth. I do feel like there are more global warming skeptics among what Eliezer called "the atheist/libertarian/technophile/sf-fan/early-adopter/programmer empirical cluster in personspace" than among, say, college professors.

In fact, very often, the uneducated position of the five year old child may be deeply flawed and the contrarian position a necessary correction to those flaws. This makes meta-contrarianism a very dangerous business.

Remember, most everyone hates hipsters.

Without meaning to imply anything about whether or not any of these positions are correct or not3, the following triads come to mind as connected to an uneducated/contrarian/meta-contrarian divide:

- KKK-style racist / politically correct liberal / "but there are scientifically proven genetic differences"
- misogyny / women's rights movement / men's rights movement
- conservative / liberal / libertarian4
- herbal-spiritual-alternative medicine / conventional medicine / Robin Hanson
- don't care about Africa / give aid to Africa / don't give aid to Africa
- Obama is Muslim / Obama is obviously not Muslim, you idiot / Patri Friedman5

What is interesting about these triads is not that people hold the positions (which could be expected by chance) but that people get deep personal satisfaction from arguing the positions even when their arguments are unlikely to change policy6 - and that people identify with these positions to the point where arguments about them can become personal.

If meta-contrarianism is a real tendency in over-intelligent people, it doesn't mean they should immediately abandon their beliefs; that would just be meta-meta-contrarianism. It means that they need to recognize the meta-contrarian tendency within themselves and so be extra suspicious and careful about a desire to believe something contrary to the prevailing contrarian wisdom, especially if they really enjoy doing so.


Footnotes

1) But what's really interesting here is that people at each level of the pyramid don't just follow the customs of their level. They enjoy following the customs, it makes them feel good to talk about how they follow the customs, and they devote quite a bit of energy to insulting the people on the other levels. For example, old money call the nouveau riche "crass", and men who don't need to pursue women call those who do "chumps". Whenever holding a position makes you feel superior and is fun to talk about, that's a good sign that the position is not just practical, but signaling related.

2) There is no need to point out just how unlikely it is that such a number is correct, nor how unscientific the survey was.

3) One more time: the fact that those beliefs are in an order does not mean some of them are good and others are bad. For example, "5 year old child / pro-death / transhumanist" is a triad, and "warming denier / warming believer / warming skeptic" is a triad, but I personally support 1+3 in the first triad and 2 in the second. You can't evaluate the truth of a statement by its position in a signaling game; otherwise you could use human psychology to figure out if global warming is real!

4) This is my solution to the eternal question of why libertarians are always more hostile toward liberals, even though they have just about as many points of real disagreement with the conservatives.

5) To be fair to Patri, he admitted that those two posts were "trolling", but I think the fact that he derived so much enjoyment from trolling in that particular way is significant.

6) Worth a footnote: I think in a lot of issues, the original uneducated position has disappeared, or been relegated to a few rednecks in some remote corner of the world, and so meta-contrarians simply look like contrarians. I think it's important to keep the terminology, because most contrarians retain a psychology of feeling like they are being contrarian, even after they are the new norm. But my only evidence for this is introspection, so it might be false.

Comment author: Yvain 12 September 2010 03:15:14PM *  2 points [-]

I found something interesting today: Dawkins/Hermione

I will be very disappointed if Methods of Rationality doesn't include some kind of explanation for this.

Comment author: CronoDAS 10 September 2010 12:24:59AM *  23 points [-]

Random thought about scope insensitivity:

One person in need is a potential ally.
A group of people in need are a potential threat.

Comment author: Yvain 10 September 2010 09:21:14AM 9 points [-]

Evidence against this as an explanation: the original studies on scope insensitivity were about birds and national parks. 20,000 birds aren't more of a threat than 200 birds, Hitchcock movies notwithstanding. Since scope insensitivity is sufficiently explained by the factors that produce it in bird and park studies, why posit extra factors to explain it on humans?

Comment author: Sniffnoy 06 September 2010 07:24:57AM *  8 points [-]

These chapters (43-46) seem to have several pieces of evidence for the "Harrymort" theory. Quirrell's reaction suggests that he recognizes the particular ideas that Harry had, which in turn suggests they're where he hid his horcruxes. That those locations also seemed obvious to Harry could be simply because they are obvious, and Voldemort used them for that same reason, but it could also indicate Harry somehow "remembering" them. That Voldemort might not have actually attempted to kill Harry after having killed Lily also suggests something may have been up there, though Voldemort may have been simply lying. And we also now have a bit of evidence that Harry's "dark side" may actually be real.

Comment author: Yvain 06 September 2010 05:19:48PM *  6 points [-]

I'm still trying to figure out what happened at Godric's Hollow.

Voldemort went in there with the intention to kill Harry (evidence: the prophecy, his seeming willingness to let Lily escape). Lily asked him to spare Harry's life in exchange for her own. It would seem that Voldemort accepted this offer in some way: he verbally agreed to the bargain, he killed Lily despite a previous intention to spare her, and Harry ended up surviving the encounter. But why would Voldemort do that when he could as easily have killed both, when he wanted Harry dead for prophecy-related reasons, and when he wanted Lily alive for Snape's sake?

Theories:

  1. Voldy had always planned to save Harry's life for his own purposes - maybe he interpreted the prophecy as meaning this would be the boy into whom he could upload his personality. He only came to Godric's Hollow to cast the personality-transfer spell onto Harry and maybe get rid of the parents. He accepted Lily's offer because it amused him to have Lily sacrifice her life when he wasn't going to kill Harry anyway.

  2. Voldy came to kill Harry, and never gave up on that intention. He pretended to accept Lily's bargain because he was Evil, and pretending to accept bargains and then breaking them is what evil people do. However, there was some hidden magic that auto-cast the Unbreakable Vow spell without Voldemort knowing. Voldemort tried to kill Harry, which broke the vow he had made to Lily, insta-killing him. Harry was Horcruxed and Voldemort's soul survived in the Horcruxes in a way relatively similar to in canon.

In response to Something's Wrong
Comment author: Yvain 06 September 2010 01:36:59PM 15 points [-]

There is a big difference between some of the examples in this post: factual issues like atheism and P=NP on one side, and political issues like Marxism and anarchism on the other. The one side we evaluate on its truth, the other side, we evaluate on its goodness.

One would hope that there is some theory that is completely true; therefore, any deviation from optimum in a theory is a genuine problem that needs to be solved. But as many commenters have said already, there isn't always a perfect solution to a political problem; a non-optimum result might still be the best option available.

It's probably a bad idea for a language to use the same words, like "right" and "wrong", to apply to both situations.

In particular, I agree with everyone who's said criticizing an optimum but imperfect social policy might be a selfish action with negative externalities. Going on about how bad it is that capitalism leaves some people poor makes the one person who does it look extra compassionate, but if everyone does it, then eventually you end up getting rid of capitalism.

So I agree with this post about factual theories but disagree when it comes to policy.

Comment author: Yvain 01 September 2010 06:53:36PM *  25 points [-]

We have not solved all your problems. Each answer only led to new questions. We are still confused - but perhaps we are confused on a higher level, and about more important things.

-- seen on a hotel bulletin board

In response to comment by Yvain on Cryonics Questions
Comment author: enoonsti 28 August 2010 06:39:47PM *  2 points [-]

By the way, I'm not here to troll, and I do have a serious question that doesn't necessarily have to do with cryonics. The goal of SIAI (Lesswrong, etc) is to learn and possibly avoid a dystopian future. If you truly are worried about a dystopian future, then doesn't that serve as a vote of "No confidence" for these initiatives?

Admittedly, I haven't looked into your history, so that may be a "Well, duh" answer :)

Comment author: Yvain 29 August 2010 07:50:51AM 8 points [-]

I suppose it serves as a vote of less than infinite confidence. I don't know if it makes me any less confident than SIAI themselves. It's still worth helping SIAI in any way possible, but they've never claimed a 100% chance of victory.

In response to Cryonics Questions
Comment author: Yvain 27 August 2010 10:57:51AM *  36 points [-]

Some of these questions, like the one about running away from a fire, ignore the role of irrational motivation.

People, when confronted with an immediate threat to their lives, gain a strong desire to protect themselves. This has nothing to do with a rational evaluation of whether or not death is better than life. Even people who genuinely want to commit suicide have this problem, which is one reason so many of them try methods that are less effective but don't activate the self-defense system (like overdosing on pills instead of shooting themselves in the head). Perhaps even a suicidal person who'd entered the burning building because e planned to jump off the roof would still try to run out of the fire. So running away from a fire, or trying to stop a man threatening you with a sword, cannot be taken as proof of a genuine desire to live, only that any desire to die one might have is not as strong as one's self-protection instincts.

It is normal for people to have different motivations in different situations. When I see and smell pizza, I get a strong desire to eat the pizza; right now, not seeing or smelling pizza, I have no particular desire to eat pizza. The argument "If your life was in immediate danger, you would want it to be preserved; therefore, right now you should seek out ways to preserve your life in the future, whether you feel like it or not" is similar to the argument "If you were in front of a sizzling piece of pizza, you would want to eat it; therefore, right now you should seek out pizza and eat it, whether you feel like it or not".

Neither argument is inevitably wrong. But first you would have to prove that the urge comes from a reflectively stable value - something you "want to want", and not just from an impulse that you "want" but don't "want to want".

The empirical reason I haven't signed up for cryonics yet is that the idea of avoiding death doesn't have any immediate motivational impact on me, and the negatives of cryonics - weirdness, costs in time and money, negative affect of being trapped in a dystopia - do have motivational impact on me. I admit this is weird and not what I would have predicted about my motivations if I were considering them in the third person, but empirically, that's how things are.

I can use my willpower to overcome an irrational motivation or lack of motivation. But I only feel the need to do that in two cases. One, where I want to help other people (eg giving to charity even when I don't feel motivated to do so). And two, when I predict I will regret my decision later (eg I may overcome akrasia to do a difficult task now when I would prefer to procrastinate). The first reason doesn't really apply here, but the second is often brought out to support cryonics signup.

Many people who signal acceptance of death appear to genuinely go peacefully and happily - that is, even to the moment of dying they don't seem motivated to avoid death. If this is standard, then I can expect to go my entire life without regretting the choice not to sign up for cryonics at any moment. After I die, I will be dead, and not regretting anything. So I expect to go all of eternity without regretting a decision not to sign up for cryonics. This leaves me little reason to overcome my inherent dismotivation to get it.

Some have argued that, when I am dead, it will be a pity, because I would be having so much more fun if I were still alive, so I ought to be regretful even though I'm not physically capable of feeling the actual emotion. But this sounds too much like the arguments for a moral obligation to create all potential people, which lead to the Repugnant Conclusion and which I oppose in just about all other circumstances.

That's just what I've introspected as the empirical reasons I haven't signed up for cryonics. I'm still trying to decide if I should accept the argument. And I'm guessing that as I get older I might start feeling more motivation to cheat death, at which point I'd sign up. And there's a financial argument that if I'm going to sign up later, I might as well sign up now, though I haven't yet calculated the benefits.

But analogies to running away from a burning building shouldn't have anything to do with it.

Comment author: Yvain 24 August 2010 01:42:01AM *  9 points [-]

I think you've hit on an important point in asking what dissociation syndromes show about the way the mind processes "selfhood", and you could expand upon that by considering a whole bunch of interesting altered states that seem to correspond to something in the temporal lobe (I can't remember the exact research).

I didn't completely follow the rest of the article. Is "consciousness" even the right term to use here? It has way too many meanings, and some of them aren't what you're talking about here - for example, I don't see why there can't be an entity that has subjective experience but no personal identity or self-knowledge. Consider calling the concept you're looking for "personal identity" instead.

I also take issue with some of the language around continuity of personal identity being an illusion. I agree with you that it probably doesn't correspond to anything in the universe, but it belongs in a category with morality of "Things we're not forced to go along with by natural law, but which are built into our goal system and finding they don't have any objective basis doesn't force us to give them up". I don't think aliens would be philosophically rash enough to stop existing just because of a belief that personal identity is an illusion.

Also, paragraph breaks!

Comment author: Yvain 21 August 2010 11:45:15PM *  18 points [-]

The bulk of this is about a vague impression that SIAI isn't transparent and accountable. You gave one concrete example of something they could improve: having a list of their mistakes on their website. This isn't a bad idea, but AFAIK GiveWell is about the only charity that currently does this, so it doesn't seem like a specific failure on SIAI's part not to include this. So why the feeling that they're not transparent and accountable?

SIAI's always done a good job of letting people know exactly how it's raising awareness - you can watch the Summit videos yourself it you want. They could probably do a bit more to publish appropriate financial records, but I don't think that's your real objection. Besides that, what? Anti-TB charities can measure how much less TB there is per dollar invested; SIAI can't measure what percentage safer the world is, since the world-saving is still in basic research phase. You can't measure the value of the Manhattan Project in "cities destroyed per year" while it's still going on.

By the Outside View, charities that can easily measure their progress with statistics like "cases of TB prevented" are better than those that can't. By the Outside View, charities that employ people who don't sound like megalomaniacal mad scientists are better than those that employ people who do. By the Outside View, charities that don't devote years of work to growing and raising awareness before really starting working full-time on their mission are better than ones that do. By the Outside View, SIAI is a sucky charity, and they know it.

There are some well-documented situations where the Outside View is superior to the Inside View, but there are also a lot of cases where it isn't - a naive Outside Viewist would have predicted Obama would've lost the election, even when he was way ahead in the polls, because by the Outside View black people don't become President. To the degree you have additional evidence, and to the degree that you trust yourself to only say you have additional evidence when you actually have additional evidence and not when you're trying to make excuses for yourself, the Inside View is superior to the Outside View. The Less Wrong Sequences are several hundred really comprehensive blog posts worth of additional evidence trying to convey Inside information on why SIAI and its strategy aren't as crazy as they sound; years of interacting with SIAI people is Inside information on whether they're honest and committed. I think these suffice to shift my probability estimates: not all the way, but preventing the apocalypse is the sort of thing one only needs a small probability to start thinking about.

The other Outside View argument would be that, whether or not you trust SIAI, it's more important to signal that you only donate to transparent and accountable organizations, in order to TDT your way into making other people only donate to transparent and accountable organizations and convince all charities to become transparent and accountable. This is a noble idea, but the world being destroyed by unfriendly AI would throw a wrench into the "improve charity" plan, so this would be an excellent time to break your otherwise reasonable rule.

Comment author: SilasBarta 04 August 2010 12:04:50PM 29 points [-]

I apologize, Yvain. Whatever reason I might have had to be suspicious did not justify me making the accusations I did, or the tone I used in the comment or surrounding discussion.

Comment author: Yvain 05 August 2010 11:06:44PM 4 points [-]

Thanks.

Comment author: Will_Sawin 03 August 2010 11:59:24PM 0 points [-]

If some of the heuristics are failing and some are succeeding, they are producing different results. Which process determines which results are correct? Should this be called "philosophical ability"?

(non-rhetorical questions)

Comment author: Yvain 05 August 2010 11:06:09PM 0 points [-]

It doesn't necessarily have to be cenralized. Some heuristics could have different weights than others, and stronger ones win out. Or there could be a reflective equilibrium among them.

...not that there's any evidence for any of this.

Comment author: timtyler 04 August 2010 10:20:43AM *  4 points [-]

Your consciousness contains the things you need to be able to reflect on in order to function properly. That seems like a much more basic way of delineating the conscious mind than the proposed signalling theory.

Yes: consciousness sometimes excludes things that it is undesirable to signal - but surely that is more of a footnote to the theory than its main feature. Quite a bit of that work is actually done by selective forgetting - which is a feature with better targeting capabilities than the filters of consciousness.

If you want the answer to involve signalling, then the ego seems like a more suitable thing to examine.

Comment author: Yvain 05 August 2010 07:40:15AM 2 points [-]

It sounds like our two theories make different predictions.

Yours suggests (if I understand right) that the function of the conscious mind is to reflect upon things. If reflecting on things is evolutionary necessary, then the conscious mind's reflections ought to be used to make decisions or something. So you predict that when a person gives reasons for eir decisions, those reasons should always be correct. It also predicts that our decisions will usually be those that would be arrived at by logical reflection.

Mine predicts that the conscious mind's reflections should be only loosely correlated with actual decisions, and more likely to be ex post facto justifications, and that even though we may have logical-sounding reasons for our decisions, on closer inspection they will look less like logic and more like the sorts of things an unconscious selfish heuristic process would arrive at.

Is that fair, or am I doing that thing where I fit the evidence to say what I want it to again?

Comment author: Yvain 05 August 2010 05:22:59AM 1 point [-]

A long time ago on Overcoming Bias, there was a thread started by Eliezer which was a link to a post on someone else's blog. The linked post posed a question, something like: "Consider two scientists. One does twenty experiments, and formulates a theory that explains all twenty results. Another does ten experiments, formulates a theory that adequately explains all ten results, does another ten experiments, and finds that eir theory correctly predicted the results. Which theory should we trust more and why?"

I remember Eliezer said he thought he had an answer to the question but was going to wait before posting it. I've since lost the post. Does anyone remember what post this is or whether anyone ever gave a really good formal answer?

Comment author: Psychohistorian 04 August 2010 01:24:36PM 5 points [-]

This feels like a worse version of epicycles, in that even if it's kind of useful, it seems like it definitely is not what's going on. The idea of lying being difficult seems to (A) presuppose a consciousness, and (B) make no sense - it seems like it would be much cheaper to evolve better lie-hiding mechanisms than to evolve consciousness. "Cognitive dissonance is adaptive with respect to expensive gestures" seems to explain pretty much all of what this theory is trying to address, without being weirdly centered on lying.

This feels like a theory that has been heavily massaged to fit facts, in the sense that your prediction of how such an individual would act seems to rather conveniently match how we actually act, rather than clearly and obviously predicting how we would act.

Comment author: Yvain 05 August 2010 05:17:20AM *  7 points [-]

If there were an elegant alternative to this model, that would be better. But I don't know of any other model that tries to comprehensively explain mental conflict without handwaving through the difficult parts.

Part of the problem may be that I assumed people already agreed with most of this model's premises. Looking at the comments, I see I was totally wrong and the average person here doesn't believe in things like the signaling theory of consciousness (which I thought was almost-universal in the Less Wrong community). So I might backtrack and try to make a sequence out of this, where I present each premise in order and slowly try to justify them. Maybe if people already believed all the premises it would look more like a reasonable way to fit a few large parts together, and less like LOOK HERE ARE TWELVE MILLION HYPOTHESIZED SYSTEMS AND SUBSYSTEMS THAT WHEN YOU PUT THEM TOGETHER KIND OF PRODUCE SOMETHING LIKE OUR BRAINS.

Comment author: cousin_it 04 August 2010 10:41:01AM *  3 points [-]

This theory seems to make a testable prediction: you will have less akrasia if your signaling requires you to reach your goal, not just show that you're working towards it. Looking at my life, I'm not sure if that's true.

Comment author: Yvain 05 August 2010 03:31:11AM *  3 points [-]

One inspiration for this line of thinking was Michael Vassar complaining about how many people volunteer to help SIAI, versus how few people actually go through with helping them once he's given them projects. I noticed I do this sort of thing all the time.

Comment author: NancyLebovitz 04 August 2010 02:12:52PM 7 points [-]

There's also a sort of akrasia which is physically based-- if I eat too much refined carbs, I can get a day or two of doing very little while thinking "I don't care, I don't care". It looks like a psychological problem, but is really well corelated with the carbs.

Comment author: Yvain 05 August 2010 03:29:04AM 2 points [-]

I agree; that's what I meant by the "wetware akrasia" footnote linked to the post about glucose.

Comment author: BenP 04 August 2010 03:50:06PM *  6 points [-]

This model is missing a plausible evolutionary explanation for how U and C may have evolved. That's a pretty gaping hole because if we don't constrain U and C to being plausible under evolution then they can be given whatever motives, responsibilities, etc. that are convenient to fit the model to existing data (see Psychohistorian's epicycle comment).

Comment author: Yvain 05 August 2010 03:27:56AM 3 points [-]

What do you mean? Do you mean the genetic explanation (as in "it was a mutation on the long arm of chromosome 20?") or do you mean the evolutionary pressures that were behind it?

I tried to explain the evolutionary pressures behind it - eg a person who couldn't signal would be unsuccessful, a person who signaled but never acted on those signals would be unconvincing. If you think those pressures wouldn't be sufficient, why not?

I don't have a play-by-play genetic explanation, but neither do a lot of well-established things.

Comment author: JanetK 04 August 2010 10:45:06PM 3 points [-]

This theory simply does not resonate with me. I do not feel that I am at all like that and neither has anyone I have known been like that. It is as off the mark as Freudian theories are, in my view. "So you wall off a little area of your mind.." Do you have any evidence for this idea that the consciousness is a walled off area?

Comment author: Yvain 05 August 2010 03:24:39AM 3 points [-]

Is your objection that you don't believe in an unconscious at all, or that you think the unconscious works in a different when than I do?

I think links in this post are sufficient to show something's* going on with mental decisions that aren't consciously available; if you haven't already, see posts like "Belief in Belief", "Cached Selves", "Simultaneously Right and Wrong", "The Apologist and the Revolutionary", et cetera. Given that unconscious processing takes place, this is an attempt to explain why and where it might work. I guess another model might be better, but I haven't really seen any other model (except Wei's and Phil's, to which this is basically a friendly amendment).

So yes, I admit it's obviously crazy, but I don't think it's crazier than necessary to explain the crazy data.

Comment author: hegemonicon 05 August 2010 12:03:36AM 1 point [-]

they honestly believe on introspection that they have admirable goals

This seems incorrect - anyone reasonably apt at introspection would not come to the conclusion "I have only admirable goals", but instead to the conclusion "I seem to have many conflicting goals". It's only a profound LACK of introspection that would make someone believe that they have only admirable goals.

Comment author: Yvain 05 August 2010 03:18:18AM 3 points [-]

Really? I honestly believe on introspection that I want to help other people and so on. I notice I'm not very good at it, but I want to.

Conflicts Between Mental Subagents: Expanding Wei Dai's Master-Slave Model

46 Post author: Yvain 04 August 2010 09:16AM

Related to: Alien Parasite Technical Guy, A Master-Slave Model of Human Preferences

In Alien Parasite Technical Guy, Phil Goetz argues that mental conflicts can be explained as a conscious mind (the "alien parasite”) trying to take over from an unsuspecting unconscious.

Last year, Wei Dai presented a model (the master-slave model) with some major points of departure from Phil's: in particular, the conscious mind was a special-purpose subroutine and the unconscious had a pretty good idea what it was doing1. But Wei said at the beginning that his model ignored akrasia.

I want to propose an expansion and slight amendment of Wei's model so it includes akrasia and some other features of human behavior. Starting with the signaling theory implicit in Wei's writing, I'll move on to show why optimizing for signaling ability would produce behaviors like self-signaling and akrasia, speculate on why the same model would also promote some of the cognitive biases discussed here, and finish with even more speculative links between a wide range of conscious-unconscious conflicts.

The Signaling Theory of Consciousness

This model begins with the signaling theory of consciousness. In the signaling theory, the conscious mind is the psychological equivalent of a public relations agency. The mind-at-large (hereafter called U for “unconscious” and similar to Wei's “master”) has socially unacceptable primate drives you would expect of a fitness-maximizing agent like sex, status, and survival. These are unsuitable for polite society, where only socially admirable values like true love, compassion, and honor are likely to win you friends and supporters. U could lie and claim to support the admirable values, but most people are terrible liars and society would probably notice.

So you wall off a little area of your mind (hereafter called C for “conscious” and similar to Wei's “slave”) and convince it that it has only admirable goals. C is allowed access to the speech centers. Now if anyone asks you what you value, C answers "Only admirable things like compassion and honor, of course!" and no one detects a lie because the part of the mind that's moving your mouth isn't lying.

This is a useful model because it replicates three observed features of the real world: people say they have admirable goals, they honestly believe on introspection that they have admirable goals, but they tend to pursue more selfish goals. But so far, it doesn't explain the most important question: why do people sometimes pursue their admirable goals and sometimes not?


Avoiding Perfect Hypocrites

In the simplest case, U controls all the agent's actions and has the ability to set C's values, and C only controls speech. This raises two problems.

First, you would be a perfect hypocrite: your words would have literally no correlation to your actions. Perfect hypocrites are not hard to notice. In a world where people are often faced with Prisoners' Dilemmas against which the only defense is to swear a pact to mutually cooperate, being known as the sort of person who never keeps your word is dangerous. A recognized perfect hypocrite could make no friends or allies except in the very short-term, and that limitation would prove fatal or at least very inconvenient.

The second problem is: what would C think of all this? Surely after the twentieth time protesting its true eternal love and then leaving the next day without so much as a good-bye, it would start to notice it wasn't pulling the strings. Such a realization would tarnish its status as "the honest one"; it couldn't tell the next lover it would remain forever true without a little note of doubt creeping in. Just as your friends and enemies would soon realize you were a hypocrite, so C itself would realize it was part of a hypocrite and find the situation incompatible with its idealistic principles.

Other-signaling and Self-Signaling

You could solve the first problem by signaling to others. If your admirable principle is to save the rainforest, you can loudly and publicly donate money to the World Wildlife Fund. When you give your word, you can go ahead and keep it, as long as the consequences aren't too burdensome. As long as you are seen to support your principles enough to establish a reputation for doing so, you can impress friends and allies and gain in social status.

The degree to which U gives permission to support your admirable principles depends on the benefit of being known to hold the admirable principle, the degree to which supporting the principle increases others' belief that you genuinely hold the principle, and the cost of the support. For example, let's say a man is madly in love with a certain woman, and thinks she would be impressed by the sort of socially conscious guy who believes in saving the rainforest. Whether or not he should donate $X to the World Wildlife Fund depends on how important winning the love of this woman is to him, how impressed he thinks she'd be to know he strongly believes in saving the rainforests, how easily he could convince her he supports the rainforests with versus without a WWF donation - and, of course, the value of X and how easily he can spare the money. Intuitively, if he's really in love, she would be really impressed, and it's only a few dollars, he would do it; but not if he's not that into her, she doesn't care much, and the WWF won't accept donations under $1000.

Such signaling also solves the second problem, the problem of C noticing it's not in control - but only partly. If you only give money when you're with a love interest and ey's standing right there, and you only give the minimum amount humanly possible so as to not repulse your date, C will notice that also. To really satisfy C, U must support admirable principles on a more consistent basis. If a stranger comes up and gives a pitch for the World Wildlife Fund, and explains that it would really help a lot of rainforests for a very low price, U might realize that C would get a little suspicious if it didn't donate at least a token amount. This kind of signaling is self-signaling: trying to convince part of your own mind.

This model modifies the original to include akrasia2 (U refusing to pursue C's goals) and the limitations on akrasia (U pursues C's goals insofar as it has to convince other people - and C itself - its signaling is genuine).

It also provides a key to explaining some superficially weird behavior. A few weeks ago, I saw a beggar on the sidewalk and walked to the other side of the street to avoid him. This isn't sane goal-directed behavior: either I want beggars to have my money, or I don't. But under this model, once the beggar asks for money, U has to give it or risk C losing some of its belief that it is compassionate and therefore being unable to convince others it is compassionate. But as long as it can avoid being forced to make the decision, it can keep both its money and C's innocence.

Thinking about this afterward, I realized how silly it was, and now I consider myself unlikely to cross the street to avoid beggars in the future. In the language of the model, C focuses on the previously subconscious act of avoiding the beggar and realizes it contradicts its principles, and so U grudgingly has to avoid such acts to keep C's innocence and signaling ability intact.

Notice that this cross-the-street trick only works if U can act without C being fully aware what happened or its implications. As we'll see below, this ability of U's has important implications for self-deception scenarios.

From Rationality to Rationalization

So far, this model has assumed that both U and C are equally rational. But a rational C is a disadvantage for U for exactly the reasons mentioned in the last paragraph; as soon as C reasoned out that avoiding the beggar contradicted its principles, U had to expend more resources giving money to beggars or lose compassion-signaling ability. If C is smart enough to realize that its principle of saving the rainforest means you ought to bike to work instead of taking the SUV, U either has to waste resources biking to work or accept a decrease in C's environmentalism-signaling ability. Far better that C never realizes it ought to bike to work in the first place.

So it's to U's advantage to cripple C. Not completely, or it loses C's language and reasoning skills, but enough that it falls in line with U's planning most of the time.

“How, in detail, does U cripple C?” is a restatement of one of the fundamental questions of Less Wrong and certainly too much to address in one essay, but a few suggestions might be in order:

- The difference between U and C seems to have a lot to do with two different types of reasoning. U seems to reason over neural inputs – it takes in things like sense perceptions and outputs things like actions, feelings, and hunches. This kind of reasoning is very powerful – for example, it can take as an input a person you've just met and immediately output a calculation of their value as a mate in the form of a feeling of lust – but it can also fail in weird ways, like outputting a desire to close a door three dozen times into the head of an obsessive-compulsive, or succumbing to things like priming. C, the linguistic one, seems to reason over propositions – it takes propositions like sentences or equations as inputs, and returns other sentences and equations as outputs. This kind of reasoning is also very powerful, and also produces weird errors like the common logical fallacies.

- When U takes an action, it relays it to C and claims it was C's action all along. C never wonders why its body is acting outside of its control; only why it took an action it originally thought it disapproved of. This relay can be cut in some disruptions of brain function (most convulsions, for example, genuinely seem involuntary), but remains spookily intact in others (if you artificially activate parts of the brain that cause movement via transcranial magnetic stimulation, your subject will invent some plausible sounding reason for why ey made that movement)3.

- C's crippling involves a tendency for propositional reasoning to automatically cede to neural reasoning and to come up with propositional justifications for its outputs, probably by assuming U is right and then doing some kind of pattern-matching to fill in blanks. For example, if you have to choose to buy one of two cars, and after taking a look at them you feel you like the green one more, C will try to come up with a propositional argument supporting the choice to buy the green one. Since both propositional and neural reasoning are a little bit correlated with common sense, C will often hit on exactly the reasoning U used (for example, if the red car has a big dent in it and won't turn on, it's no big secret why U's heuristics rejected it) but in cases where U's justification is unclear, C will end up guessing and may completely fail to understand the real reasons behind U's choice. Training in luminosity can mitigate this problem, but not end it.

- A big gap in this model is explaining why sometimes C openly criticizes U, for example when a person who is scared of airplanes says “I know that flying is a very safe mode of transportation and accidents are vanishingly unlikely, but my stupid brain still freaks out every time I go to an airport”. This might be justifiable along the lines that allowing C to signal that it doesn't completely control mental states is less damaging than making C look like an idiot who doesn't understand statistics – but I don't have a theory that can actually predict when this sort of criticism will or won't happen.

- Another big gap is explaining how and when U directly updates on C's information. For example, it requires conscious reasoning and language processing to understand that a man on a plane holding a device with a countdown timer and shouting political and religious slogans is a threat, but a person on that plane would experience fear, increased sympathetic activation, and other effects mediated by the unconscious mind.

This part of the model is fuzzy, but it seems safe to assume that there is some advantage to U in changing C partially, but not completely, from a rational agent to a rubber-stamp that justifies its own conclusions. C uses its propositional reasoning ability to generate arguments that support U's vague hunches and selfish goals.

How The World Would Look

We can now engage, with a little bit of cheating, in some speculation about how a world of agents following this modified master-slave model would look.

You'd claim to have socially admirable principles, and you'd honestly believe these claims. You'd pursue these claims at a limited level expected by society: for example, if someone comes up to you and asks you to donate money to children in Africa, you might give them a dollar, especially if people are watching. But you would not pursue them beyond the level society expects: for example, even though you might consciously believe saving a single African child (estimated cost: $900) is more important than a plasma TV, you would be unlikely to stop buying plasma TVs so you could give this money to Africa. Most people would never notice this contradiction; if you were too clever to miss it you'd come up with some flawed justification; if you were too rational to accept flawed justifications you would just notice that it happens, get a bit puzzled, call it “akrasia”, and keep doing it.

You would experience borderline cases, where things might or might not be acceptable, as moral conflicts. A moral conflict would feel like a strong desire to do something, fighting against the belief that, if you did it, you would be less of the sort of person you want to be. In cases where you couldn't live with yourself if you defected, you would cooperate; in cases where you could think up any excuse at all that allowed you to defect and still consider yourself moral, you would defect.

You would experience morality not as a consistent policy to maximize utility across both selfish and altruistic goals, but as a situation-dependent attempt to maximize feelings of morality, which could be manipulated in unexpected ways. For example, as mentioned before, going to the opposite side of the street from a beggar might be a higher-utility option than either giving the beggar money or explicitly refusing to do so. In situations where you were confident in your morality, you might decide moral signaling was an inefficient use of resources – and you might dislike people who would make you feel morally inferior and force you to expend more resources to keep yourself morally satisfied.

Your actions would be ruled by “neural reasoning” that outputs expectations different from the ones your conscious reasoning would endorse. Your actions might hinge on fears which you knew to be logically silly, and your predictions might come from a model different from the one you thought you believed. If it was necessary to protect your signaling ability, you might even be able to develop and carry out complicated plots to deceive the conscious mind.

Your choices would be determined by illogical factors that influenced neural switches and levers and you would have to guess at the root causes of your own decisions, often incorrectly – but would defend them anyway. When neural switches and levers became wildly inaccurate due to brain injury, your conscious mind would defend your new, insane beliefs with the same earnestness with which it defended your old ones.

You would be somewhat rational about neutral issues, but when your preferred beliefs were challenged you would switch to defending them, and only give in when it is absolutely impossible to keep supporting them without looking crazy and losing face.

You would look very familiar.

 

Footnotes

1. Wei Dai's model gets the strongest compliment I can give: after reading it, it seemed so obvious and natural to think that way that I forgot it was anyone's model at all and wrote the first draft of this post without even thinking of it. It has been edited to give him credit, but I've kept some of the terminology changes to signify that this isn't exactly the same. The most important change is that Wei thinks actions are controlled by the conscious mind, but I side with Phil and think they're controlled by the unconscious and relayed to the conscious. The psychological evidence for this change in the model are detailed above; some neurological reasons are mentioned in the Wegner paper below.

2. Or more accurately one type of akrasia. I disagreed with Robin Hanson and Bryan Caplan when they said a model similar to this explains all akrasia, and I stand by that disagreement. I think there are at least two other, separate causes: akrasia from hyperbolic discounting, and the very-hard-to-explain but worthy-of-more-discussion-sometime akrasia from wetware design.

3. See Wegner, "The Mind's Best Trick: How We Experience Conscious Will" for a discussion of this and related scenarios.

Comment author: AlanCrowe 03 August 2010 11:12:26AM 7 points [-]

This belongs on the parody site http://morewrong.com. Please build it :-)

Comment author: Yvain 03 August 2010 10:13:43PM 15 points [-]

I don't know; the more Less Wrong I read, the more I start to think Lovecraft was on to something.

Delving too far in our search for knowledge is likely to awaken vast godlike forces which are neither benevolent nor malevolent but horrifyingly indifferent to humanity. Some of these forces may be slightly better or worse than others, but all of them could and would swat our civilization away like a mosquito. Such forces may already control other star systems.

The only defense against such abominations is to study the arcane knowledge involved in summoning or banishing these entities; however, such knowledge is likely to cause its students permanent psychological damage or doom them to eternities of torture.

Comment author: Matt_Simpson 03 August 2010 09:06:48AM *  0 points [-]

I'm not familiar enough with Pearl's formalism to really understand TDT - or at least that's why I haven't really dove into TDT yet. I'd love to hear why you think Kant sounds more like TDT though. I'm suspecting it has something to do with considering counterfactuals.

Comment author: Yvain 03 August 2010 09:28:48AM *  1 point [-]

I'm not familiar at all with Pearl's formalism. But from what I see on this site, I gather that the key insight of updateless decision theory is to maximize utility without conditioning on information about what world you're in, and the key insight of timeless decision theory is what you're describing (Eliezer summarizes it as "Choose as though controlling the logical output of the abstract computation you implement, including the output of all other instantiations and simulations of that computation.")

Comment author: ewbrownv 02 August 2010 04:27:40PM 12 points [-]

My impression is that in MoR Voldemort was a passionate young revolutionary in the first war, but since then he's gotten older and his outlook has changed. He sees the muggles as the greatest threat now, and he recognizes that his history means he can't take power in his own name without a long and devastating series of wars that would leave the magical world exhausted and vulnerable to this outside threat. So it would be rather convenient if he could sway Harry to his way of thinking and arrange for the wizarding world to unite under a leader who sees him as a mentor...

Comment author: Yvain 03 August 2010 08:22:39AM *  7 points [-]

I agree with your analysis.

But gwern's description of Harry's victory over Voldemort as a "black swan" doesn't satisfy me. The canon explanation - that the Power of Love auto-defeats all dark magic, and either no one had ever noticed this before, or Voldemort just assumed no one would use that strategy despite its obvious game-winning power because Evil Cannot Comprehend Good - doesn't seem like the kind of thing that would cut it in Methods.

One remote possibility is that Voldemort realized he'd inspired so much hatred that he'd never be able to unite the magical world without first breaking its power so badly it would be useless to him, so he found a kid with Dark Lord potential, stuck enough of his soul into him that he felt in control, and then faked his own death in such a way as to make his chosen heir the sort of hero whom everyone would rally around. This is probably too complicated for a smart Slytherin who'd seen The Tragedy of Light to try, but there's got to be some sort of weird explanation for why Voldemort lost ten years ago, and why he lost to a kid with precisely the sort of plotting ability and mastery of Muggle methodology Voldy needs for his plots.

Comment author: Matt_Simpson 02 August 2010 05:02:37PM *  8 points [-]

Was Kant implicitly using UDT?

Consider Kant's categorical imperative. It says, roughly, that you should act such that you could will your action as a universal law without undermining the intent of the action. For example, suppose you want to obtain a loan for a new car and never pay it back - you want to break a promise. In a world where everyone broke promises, the social practice of promise keeping wouldn't exist and thus neither would the practice of giving out loans. So you would undermine your own ends and thus, according to the categorical imperative, you shouldn't get a loan without the intent to pay it back.

Another way to put Kant's position would be that you should choose such that you are choosing for all other rational agents. What does UDT tell you to do? It says (among other things) that you should choose such that you are choosing for every agent running the same decision algorithm as yourself. It wouldn't be a stretch to call UDT agents rational. So Kant thinks we should be using UDT! Of course, Kant can't draw the conclusions he wants to draw because no human is actually using UDT. But that doesn't change the decision algorithm Kant is endorsing.

Except... Kant isn't a consequentialist. If the categorical imperative demands something, it demands it no matter the circumstances. Kant famously argued that lying is wrong, period. Even if the fate of the world depends on it.

So Kant isn't really endorsing UDT, but I thought the surface similarity was pretty funny.

Comment author: Yvain 03 August 2010 07:39:32AM 1 point [-]

I thought Kant sounded a lot more like TDT than UDT. Or was that what you meant?

Comment author: Wei_Dai 28 July 2010 08:15:24AM *  1 point [-]

So the question is not "why don't we have any self-shadowing blind-spots", it is "why do we have a nontrivial set of non-self-shadowing blind spots?"

Agreed, but I think it's also, "why do we have fewer self-shadowing blind-spots than we might expect, given what we know about how evolution works?"

And while you're right that we can't be sure at this point that we have zero self-shadowing blind-spots (philosophical oversights that we'll never detect), I think there's a reasonable chance that's in fact the case. ETA: My argument for this belief is that one possible reason why we have fewer self-shadowing blind-spots than we might expect is that there is a single "ability to philosophize" that is sufficient (given enough raw intelligence) to overcome all such blind spots.

Comment author: Yvain 28 July 2010 08:18:13PM 3 points [-]

The opposite explanation also works: we use so many unrelated heuristics that there's no single area where they all fail simultaneously.

Book Review: The Root of Thought

47 Post author: Yvain 22 July 2010 08:58AM

Related to: Brain Breakthrough! It's Made of Neurons!

I can't really recommend Andrew Koob's The Root of Thought. It's poorly written, poorly proofread, lacking much more information than is in the Scientific American review, and comes across as about one part neuroscience to three parts angry rant. But it does present an interesting hypothesis and an interesting case study on a major failure of rationality.

Only about ten percent of the brain is made of neurons; the rest is a diverse group of cells called "glia". "Glia" is Greek for glue, because the scientists who discovered them decided that, since they were in the brain and they weren't neurons, they must just be there to glue the neurons together. Since then, new discoveries have assigned glial cells functions like myelination, injury repair, immune defense, and regulation of blood flow: all important, but mostly things only a biologist could love. The Root of Thought argues that glial cells, especially a kind called astrocytes, are also important in some of the higher functions of thought, including memory, cognition, and maybe even creativity. This is interesting to neuroscientists, and the story of how it was discovered is also interesting to us as aspiring rationalists.

Glial cells involved in processing

Koob's evidence is indirect but suggestive. He points out that more intelligent animals have a higher astrocyte to neuron ratio than less intelligent animals, all the way from worms with one astrocyte per thirty neurons, to humans with an astrocyte: neuron ratio well above one. Within the human brain, the areas involved in higher thought, like the cortex, are the ones with the highest astrocyte:neuron ratio, and the most down-to-earth, like the cerebellum, have barely any astrocytes at all. Especially intelligent humans may have higher ratios still: one of the discoveries made from analyzing Einstein's brain was that he had an unusually large number of astrocytes in the part of his brain responsible for mathematical processing. And learning is a stimulus for astrocyte development. When canaries learn new songs, new astrocytes grow in the areas responsible for singing.

Astrocytes have a structure especially suited for learning and cognition. They have their own gliotransmitters, similar in function to neurotransmitters, and they communicate with one another, sparking waves of astrocyte activity across areas of the brain. Like neurons, they can enter an active state after calcium release, but unlike neurons, which get calcium only when externally activated, astrocytes can fill with calcium either because of external stimuli or when their own calcium stores randomly leak out into the cell, a process which resembles the random, unprovoked nature of thought during sensory deprivation and dreaming.

Astrocytes also affect and are affected by neurons. Each astrocyte "monitors" thousands of synapses, and releases calcium based on the input it receives. Output from astrocytes, in turn, affects the behavior of neurons. Astrocytes can take up or break down neurotransmitters, which changes the probability of nearby neurons activating, and they can alter synapses, promoting some and pruning others in a process likely linked to long-term potentiation in the brain.

Although it wasn't in the book, very recent research shows a second type of glial cell, the immune-linked microglia, play a role in behavior that may be linked to obsessive-compulsive disorder; a microglia-altering bone marrow transplant cures an OCD-like disease in mice.

By performing computations that influence the firing of neurons, glial cells at the very least play a strong supporting role in cognition. Koob goes way beyond that (and really beyond what he can support) and argues that actually neurons play a supporting role to glia, being little more than the glorified wires that relay astroglial commands. His argument is very speculative and uses words like "could" a lot, but the evidence at least shows that glia are more important than a century of neurology has given them credit for.


We don't know how much we don't know about cognitive science

Previous Less Wrong articles, for example Artificial Addition, have warned against trying to replicate a process without understanding it by copying a few surface features. One of the most popular such ideas is to replicate the brain by copying the neurons and seeing what happens. For example, IBM's Blue Brain project hopes to create an entire human brain by modeling it neuron for neuron, without really understanding why brains work or why neurons do what they do1.

We've made a lot of progress in cognitive science in the past century. We know where in the brain various activities take place, we know the mechanisms behind some of the more easily studied systems like movement and perception, and we've started researching the principles of intelligence that the brain must implement to do what it does. It's tempting to say that we more or less understand the brain, and the rest is just details. One of the take-home messages from this book is that, although cognitive scientists can justifiably be proud of their progress, our understanding still hasn't even met the low bar of being entirely sure we're even studying all the right kinds of cells, and this calls into question our ability to meet the higher bar of being able to throw what we know into a simulator and hope it works itself out.

A horrible warning about community irrationality

In the late 19th century, microscopy advanced enough to look closely at the cellular structure of the brain. The pioneers of neurology decided that neurons were interesting and glia were the things you had to look past to get to the neurons. This theory should have raised a big red flag: Why would the brain be filled with mostly useless cells? But for about seventy five years, from the late 19th century to the mid to late 20th, no one seriously challenged the assumption that glia played a minor role in the brain.

Koob attributes the glia's image problem to the historical circumstances of their discovery. Neurons are big, peripherally located, and produce electrical action potentials. This made them both easy to study and very interesting back in the days when electricity was the Hot New Thing. Scientists first studied neurons in the periphery, got very excited about them, and later followed them into the brain, which turned out to be a control center for all the body's neurons. This was interesting enough that neurologists, people who already had thriving careers in the study of neurons, were willing to overlook the inconvenient presence of several other types of cells in the brain, which they relegated to a supporting role. The greatest of these early pioneers of neurology, Santiago Ramon y Cajal, was the brother of the neurologist who first proposed the idea that glial cells functioned as glue and may have (Koob theorizes) let familial loyalty influence his thinking. The community took his words as dogma and ignored glia for years, a choice no doubt made easier by all the exciting discoveries going on around neurons. Koob discussed the choice facing neuroscientists in the early 20th century: study the cell that seemed on the verge of yielding all the secrets of the human mind, or tell your advisor you wanted to study glue instead. Faced with that decision, virtually everyone chose to study the neurons.

There wasn't any sinister cabal preventing research into glia. People just didn't think of it. Everyone knew that neurons were the only interesting type of cell in the brain. They assumed that if there was some other cell that was much more common and also very important, somebody would have noticed. I've read neuroscience books, I read the couple of paragraphs where they mentioned glial cells, and I shrugged and kept reading, because I assumed if they were hugely important somebody would have noticed.

The heuristic, that an entire community doesn't just miss low-hanging fruit, is probably a good one and as many people have pointed out the vast majority of people who think they've found something that the scientific community has missed are somewhere between wrong and crackpot. Science is usually pretty good at finding and recognizing its mistakes, and even in the case of glial cells they did eventually find and recognize the mistake. It just took them a century.

One common theme across Less Wrong and SIAI is that there are some relatively little-known issues that, upon a moderate amount of thought, seem vitally important. And one of the common arguments against this theme is that if this were true, surely somebody would have noticed. The lesson of glial cells is that sometimes this just doesn't happen.

Related: Glial Cells: Their Role In Behavior, Underappreciated Star-Shaped Cells May Help Us Breathe, Glial Cells Aid Memory Formation, New Role For Supporting Brain Cells, Support Cells, Not Neurons, Lull Brain To Sleep

Comment author: Yvain 10 July 2010 10:51:59PM *  4 points [-]

The reason Royal Navy [nuclear missles] can be launched without a code is that when it was suggested failsafes should be introduced the British Admiralty took insult at the implication that Officers of the Royal Navy would ever consider launching nuclear missiles without orders or unless it was the correct thing to do.

-- TV Tropes, A Nuclear Error

Comment author: VNKKET 01 July 2010 10:07:28PM *  9 points [-]

This is a mostly-shameless plug for the small donation matching scheme I proposed in May:

I'm still looking for three people to cross the "membrane that separates procrastinators and helpers" by donating $60 to the Singularity Institute. If you're interested, see my original comment. I will match your donation.

Comment author: Yvain 02 July 2010 02:10:11AM 4 points [-]

Done!

Comment author: VNKKET 24 May 2010 11:27:18PM *  7 points [-]

ETA: This scheme is done. All three donations have been made and matched by me.

I want to give $180 to the Singularity Institute, but I'm looking for three people to match my donation by giving at least $60 each. If this scheme works, the Singularity Institute will get $360.

If you want to become one of the three matchers, I would be very grateful, and here's how I think we should do it:

  1. You donate using this link. Reply to this thread saying how much you are donating. Feel free to give more than $60 if you can spare it, but that won't affect how much I give.

  2. In your donation's "Public Comment" field, include both a link to your reply to this thread and a note asking for a Singularity Institute employee to kindly follow that link and post a response saying that you donated. ETA: Step 2 didn't work for me, so I don't expect it to work for you. For now, I'll just believe you if you say you've donated. If you would be convinced to donate by seeing evidence that I'm not lying, let me know and I'll get you some.

  3. I will do the same. (Or if you're the first matching donor, then I already have -- see directly below.)

To show that I'm serious, I'm donating my first $60 right now. I will donate my second $60 after the second matching donor, and my third $60 after the third matching donor.

If you already donate regularly, please wait until it looks like my scheme is failing before taking up one of the matching-donor slots. But if you have never donated despite always wanting to, then here's a chance to double your help.

I'm also interested in information people might have about whether this scheme is a good idea (compared to, say, quietly making the donation alone).

Comment author: Yvain 02 July 2010 02:09:32AM *  2 points [-]

I've donated $60 and put a message requesting confirmation here in my public comment.

Comment author: gwern 07 June 2010 09:20:11PM *  8 points [-]

I would take Rick Cook's approach - look for meta-spells and figure out how to combine them into something Turing-complete. From canon, we already know that spells can operate on spells ('priori incantatem' or something like that), and I'm almost sure that some spells do logical operations.

If that doesn't work out, start making the Philosopher's Stone. I will know that it's possible, and that's half the battle. Once I have the Stone, then the question of 'fastest method to omnipotence' loses its urgency.

(If this is simply not possible for a 1st Year, then I will set my sights lower on the felix/luck potion; Harry has enough money to finance all the ingredients he could possibly waste, and once you have a vat of luck potion, you can spend it on research in the library, random generation of possible recipes, or direct attempts at creating the Stone.)

Comment author: Yvain 07 June 2010 09:29:06PM 28 points [-]

Really, you should use it to try to discover a more powerful luck potion, then take the more powerful luck potion to try to discover a more powerful luck potion still, until eventually you get a hard-takeoff scenario where ever-more-powerful luck potions are falling from the sky into your hands by pure chance every second.

After the luck-ularity, Harry can just throw a random rock up in the air, and it will hit Lord Voldemort right between the eyes, killing him instantly at the same time the Pioneer probe crashes into an asteroid.

Comment author: JoshuaZ 07 June 2010 07:07:20PM 3 points [-]

What does Fallacyzilla have on its chest? It looks like it has "A -> B, ~B, therefore ~A" But that is valid logic. Am I misreading it or did you mean to put "A -> B, ~A, therefore ~B"? That would be actually wrong.

Comment author: Yvain 07 June 2010 07:13:23PM *  8 points [-]

I noticed that two seconds after I put it up and it's now corrected...er...incorrected. (Today I learned - my brain has that same annoying auto-correct function as Microsoft Word)

Comment author: RichardKennaway 07 June 2010 02:28:06PM 4 points [-]

I'd like to see a picture of this LW cannon!

Comment author: Yvain 07 June 2010 07:02:37PM 16 points [-]

I'd like to see a picture of this LW cannon!

Rather than waste time doing both your cannon request and Roko's Fallacyzilla request, I just combined them into one picture of the Less Wrong Cannon attacking Fallacyzilla.

...now someone take Photoshop away from me, please.

Comment author: DanArmak 07 June 2010 04:58:17PM 0 points [-]

I still assume that doctors actually want to help people

Homeopathy is at best a placebo. It's rare that there's no better medical way to help someone. Your assumption is counter to the facts.

Certainly doctors want to help people - all else being equal. But if they practice homeopathy extensively, then they are prioritizing other things over helping people.

If the market condition (i.e. the patients' opinions and desires) are such that they will not accept scientific medicine, and will only use homeopathy anyway, then I suggest then the best way to help people is for all doctors to publicly denounce homeopathy and thus convince at least some people to use better-than-placebo treatments instead.

Comment author: Yvain 07 June 2010 06:54:04PM *  6 points [-]

Homeopathy is at best a placebo. It's rare that there's no better medical way to help someone.

I disagree - at least with the part about "it's rare that there's no better medical way to help people". It's depressingly common that there's no better medical way to help people. Things like back pain, tiredness, and muscle aches - the commonest things for which people see doctors - can sometimes be traced to nice curable medical reasons, but very often as far as anyone knows they're just there.

Robin Hanson has a theory - and I kind of agree with him - that homeopathy fills a useful niche. Placebos are pretty effective at curing these random (and sometimes imagined) aches and pains. But most places consider it illegal or unethical for doctors to directly prescribe a placebo. Right now a lot of doctors will just prescribe aspirin or paracetamol or something, but these are far from totally harmless and there are a lot of things you can't trick patients into thinking aspirin is a cure for. So what would be really nice, is if there was a way doctors could give someone a totally harmless and very inexpensive substance like water and make the patient think it was going to cure everything and the kitchen sink, without directly lying or exposing themselves to malpractice allegations.

Where this stands or falls is whether or not it turns patients off real medicine and gets them to start wanting homeopathy for medically known, treatable diseases. Hopefully it won't - there aren't a lot of people who want homeopathic cancer treatment - but that would be the big risk.

Comment author: Wei_Dai 05 June 2010 05:22:11PM 8 points [-]

"I am precommiting that anyone who cloned themselves a trillion times gets all their clones killed. This precommitment will prevent anyone who genuinely understands my source code from having cloned themselves in the past, and will therefore increase utility."

Wait, increase utility according to what utility function? If it's an aggregate utility function where Dr. Evil has 99% weight, then why would that precommitment increase utility?

Comment author: Yvain 05 June 2010 08:25:12PM 5 points [-]

You're right. It will make a commitment to stop anyone who tries the same thing later, but it won't apply it retroactively. The original comment is wrong.

Comment author: Yvain 05 June 2010 02:35:43PM *  4 points [-]

EDIT: Doesn't work, see Wei Dai below.

This isn't a bug in CEV, it's a bug in the universe. Once the majority of conscious beings are Dr. Evil clones, then Dr. Evil becomes a utility monster and it gets genuinely important to give him what he wants.

But allowing Dr. Evil to clone himself is bad; it will reduce the utility of all currently existing humans except Dr. Evil.

If a normal, relatively nice but non-philosopher human ascended to godhood, ve would probably ignore Dr. Evil's clones' wishes. Ve would destroy the clones and imprison the doctor, because ve was angry at Dr. Evil for taking the utility-lowering action of cloning himself and wanted to punish him.

But everything goes better than expected! Dr. Evil hears a normal human is ascending to godhood, realizes making the clones won't work, and submits passively to the new order. And rationalists should win, so a superintelligent AI should be able to do at least as well as a normal human by copying normal human methods when they pay off.

So an AI with sufficiently good decision theory could (I hate to say "would" here, because making quick assumptions that an AI would do the right thing is a good way to get yourself killed) use the same logic. Ve would say, before even encountering the world "I am precommiting that anyone who cloned themselves a trillion times gets all their clones killed. This precommitment will prevent anyone who genuinely understands my source code from having cloned themselves in the past, and will therefore increase utility." Then ve opens ver sensors, sees Dr. Evil and his clones, and says "Sorry, I'd like to help you, but I precommited to not doing so," kills all of the clones as painlessly as possible, and get around to saving the world.

Comment author: ata 05 June 2010 01:54:50AM *  7 points [-]

You are magnificent.

(Alternate title for the LW tabloid — "The Rational Enquirer"?)

In response to comment by ata on Open Thread: June 2010
Comment author: Yvain 05 June 2010 10:28:59AM 1 point [-]

That's....brilliant. I might have to do another one just for that title.

Comment author: Yvain 02 June 2010 06:51:02PM 2 points [-]

Is Randi a singularitarian?

Comment author: Yvain 01 June 2010 11:17:06PM *  43 points [-]

Cleaning out my computer I found some old LW-related stuff I made for graphic editing practice. Now that we have a store and all, maybe someone here will find it useful:

Comment author: MichaelVassar 31 May 2010 04:07:31PM *  3 points [-]

Great Post!

Anyway, on to the obligatory quibble. "throwing biological solutions at spiritual problems might be disrespectful or dehumanizing, or a band-aid that doesn't affect the deeper problem" The 6 criteria for disease, including 'biological' in so far as that means caused by biological processes simple enough to understand relatively easily and confidently, do seem to me to each provide weak evidential support for any given treatment not being disrespectful, dehumanizing, or superficial. They also seem to provide weak evidence against the listed "reasonable objections to treating any condition with drugs". None of the criteria are individually or collectively decisive on those points, but it does seem to me like you could find correlations if you looked.

Comment author: Yvain 31 May 2010 10:01:22PM 6 points [-]

You'll have to explain that more. I would have said that "dehumanizing" and "disrespectful" are meaningless weasel words in the context of someone freely choosing to take a drug. Disrespect needs a victim, and I'm wary of the idea of being disrespectful to yourself.

Comment author: colah 31 May 2010 05:58:51PM 8 points [-]

Perhaps I'm misunderstanding, but

There are several very reasonable objections to treating any condition with drugs, whether it be a classical disease like cancer or a marginal condition like alcoholism. The drugs can have side effects. They can be expensive. They can build dependence. They may later be found to be placebos whose efficacy was overhyped by dishonest pharmaceutical advertising.. They may raise ethical issues with children, the mentally incapacitated, and other people who cannot decide for themselves whether or not to take them. But these issues do not magically become more dangerous in conditions typically regarded as "character flaws" rather than "diseases", and the same good-enough solutions that work for cancer or heart disease will work for alcoholism and other such conditions.

seems to summarise to:

(1) Medical treatments (drugs, surgery, et cetera) for conditions that can be treated in other ways can have negative consequences. (2) But so do those for conditions without other treatments and we use those. (3) Therefore: we should not object to these treatments on the grounds of risks.

I'd question the validity of this argument. Consider a scenario where there are two treatments for a condition: A and B. A has lower risks than B. Where is the flaw in the following argument:

(1) Treating the condition with B has risks. (2) But the treatments used for other conditions have similar risks. (3) Therefore: we should not object to B on the grounds of risks.

The problem with the argument is that it draws a false analogy between this condition (where there is a lower and higher risk treatment) and others where the only treatment is of similar risk to the high risk treatment for this condition.

I'm not saying the people with conditions like obesity shouldn't get medical treatment: there are compelling advantages to it, such as the decreased amount of effort involved and faster progress... But I think that this argument isn't valid.

Comment author: Yvain 31 May 2010 09:58:41PM 7 points [-]

If I understand you right, you're saying that allowing drugs might discourage people from even trying the willpower-based treatments, which provides a cost of allowing drugs that isn't present in diseases without a willpower-based option.

It's a good point and I'm adding it to the article.

Comment author: soreff 31 May 2010 07:01:16PM *  2 points [-]

Very good article!

A couple of comments:

So here, at last, is a rule for which diseases we offer sympathy, and which we offer condemnation: if giving condemnation instead of sympathy decreases the incidence of the disease enough to be worth the hurt feelings, condemn; otherwise, sympathize.

Almost agreed: It is also important to recheck criterion 4:

Something unpleasant; when you have it, you want to get rid of it

to see if reducing the incidence of the disease is actually a worthwhile goal.

On another note:

Cancer satisfies every one of these criteria, and so we have no qualms whatsoever >about classifying it as a disease.

Criterion 3:

Something rare; the vast majority of people don't have it

is somewhat arguable, at least for some types. Quoth Wikipedia

Autopsy studies of Chinese, German, Israeli, Jamaican, Swedish, and Ugandan men who died of other causes have found prostate cancer in thirty percent of men in their 50s, and in eighty percent of men in their 70s

Comment author: Yvain 31 May 2010 09:54:55PM 2 points [-]

Good points. But prostate cancer might be an "ostrich" version of cancer (see the link on "ostrich" above) and something like breast cancer might be considered more like a type specimen.

Comment author: Vladimir_M 31 May 2010 01:07:18AM *  8 points [-]

So if there existed a hypothetical institution with the power to mete out preventive imprisonment, and which would reliably base its decisions on mathematically sound consequentialist arguments, would you be OK with it? I'm really curious how many consequentialists here would bite that bullet. (It's also an interesting question whether, and to what extent, some elements of the modern criminal justice system already operate that way in practice.)

[EDIT: To clarify a possible misunderstanding: I don't have in mind an institution that would make accurate predictions about the future behavior of individuals, but an institution that would preventively imprison large groups of people, including many who are by no means guaranteed to be future offenders, according to criteria that are accurate only statistically. (But we assume that they are accurate statistically, so that its aggregate effect is still evaluated as positive by your favored consequentialist calculus.)]

This seems to be the largest lapse of logic in the (otherwise very good) above post. Only a few paragraphs above an argument involving the reversal test, the author apparently fails to apply it in a situation where it's strikingly applicable.

Comment author: Yvain 31 May 2010 09:49:02PM 7 points [-]

I'll bite that bullet. I already have in the case of insane people and arguably the case terrorists who belong to a terrorist cell and are hatching terrorist plots but haven't committed any attacks yet.

But it would have to be pretty darned accurate, and there would have to be a very low margin of error.

Comment author: SilasBarta 31 May 2010 06:05:42PM *  12 points [-]

I remember being in a similar argument myself. I was talking with someone about how I had (long ago!) deliberately started smoking to see if quitting would be hard [1], and I found that, though there were periods where I'd had cravings, it wasn't hard to distract myself, and eventually they went away and I was able to easily quit.

The other person (who was not a smoker and so probably didn't take anything personally) said, "Well, sure, in that case it's easy to quit smoking, because you went in with the intent to prove it's easy to quit. Anyone would find it easy to stay away from cigarettes in that case!"

So I said, "Then shouldn't that be the anti-smoking tactic that schools use? Make all students take up smoking, just to prove they can quit. Then, everyone will grow up with the ability to quit smoking without much effort."

[1] and many, many people have told me this is insane, so no need to remind me

Comment author: Yvain 31 May 2010 09:44:56PM 12 points [-]

I met someone who started smoking for the same reason you did once and is still addicted, so you couldn't have been at that much of an advantage.

I am torn between telling you you're insane and suggesting you take up crack on a sort of least convenient possible world principle.

Comment author: Nick_Tarleton 31 May 2010 02:30:48AM *  8 points [-]

Sort-of nitpick:

The consequentialist model of blame is very different from the deontological model. Because all actions are biologically determined, none are more or less metaphysically blameworthy than others, and none can mark anyone with the metaphysical status of "bad person" and make them "deserve" bad treatment. Consequentialists don't on a primary level want anyone to be treated badly, full stop; thus is it written: "Saddam Hussein doesn't deserve so much as a stubbed toe." But if consequentialists don't believe in punishment for its own sake, they do believe in punishment for the sake of, well, consequences.

I would say "utilitarians" rather than "consequentialists" here; while both terms are vague, consequentialism is generally more about the structure of your values, and there's no structural reason a consequentialist (/ determinist) couldn't consider it desirable for blameworthy people to be punished. (Or, with regard to preventative imprisonment of innocents, undesirable for innocents to be punished, over and above the undesirability of the harm that the punishment constitutes.)

Comment author: Yvain 31 May 2010 09:42:51PM 7 points [-]

I installed a mental filter that does a find and replace from "utilitarian" to "consequentialist" every time I use it outside very technical discussion, simply because the sort of people who don't read Less Wrong already have weird and negative associations with "utilitarian" that I can completely avoid by saying "consequentialist" and usually keep the meaning of whatever I'm saying intact.

Less Wrong does deserve better than me mindlessly applying that filter. But you'd need a pretty convoluted consequentialist system to promote blame (and if you were willing to go that far, you could call a deontologist someone who wants to promote states of the world in which rules are followed and bad people are punished, and therefore a consequentialist at heart). Likewise, you could imagine a preference utilitarian who wants people to be punished just because e or a sufficient number of other people prefer it. I'm not sufficiently convinced enough to edit the article, though I'll try to be more careful about those terms in the future.

Comment author: sketerpot 31 May 2010 03:14:03AM *  12 points [-]

Someone once quipped about a Haskell library that "You know it's a good library when just reading the manual removes the problem it solves from your life forever." I feel the same way about this article. That's a compliment, in case you were wondering.

The one criticism I would make is that it's long, and I think you could spread this to other sites and enlighten a lot of people if you wrote an abridged version and perhaps illustrated it with silly pictures of cats.

Comment author: Yvain 31 May 2010 09:34:39PM 12 points [-]

Thank you very much. That's exactly the feeling I hoped people would have if this dissolved the question and it's great to hear.

I can't think of how to make this shorter without removing content (especially since this is already pitched at an advanced audience - anything short of LW and I'd have to explain status quo biases, preference reversal tests, and actually justify determinism).

I can, however, give you an lolcat if you want one.

Comment author: [deleted] 31 May 2010 12:35:44PM 3 points [-]

I like this because it dissolves the question quite effectively. I'm not sure the question should be dissolved, though ... what about the sister?

This is why I'm not a consequentialist all the way. We may regard it as obvious that cancer is undesirable, but there really may be some who disagree. There are some who disagree that obesity is undesirable. There are some who disagree that depression is undesirable. Health is one issue where most people (in our society) are particularly unlikely to take account of differences in opinion.

Praise and blame are not just alternate possible ways to treat a disease. Example: I personally think obesity is undesirable. If I know an obese person who's happy that way, though, I wouldn't dream of trying to "treat" her, because it's none of my business. Yet I'm still curious to what extent she's "blameworthy" or personally responsible. Judging someone's blameworthiness or praiseworthiness doesn't necessarily result in trying to improve her behavior; it has to do with what opinion I hold of her.

That's a libertarian deontologist view, yeah, but it's close enough to ordinary behavior that I think we should consider whether it's completely unreasonable.

Comment author: Yvain 31 May 2010 09:33:05PM 4 points [-]

I sort of agree. I didn't treat this issue because the post was already getting too long.

We have various incentives to want obese people to become thin: paternalistic concern for their health, negative externalities, selfish reasons if we're their friend or relative and want to continue to enjoy their company without them dying early, aesthetic reasons, the emotional drain of offering them sympathy if we don't think they deserve it. One of the most important reasons is helping them overcome akrasia - if they want to become thinner, us being seen to condemn obesity might help them.

If they don't want to become thinner, that incentive goes away. The other incentives might or might not be enough to move us on their own.

(usually, though, these things only become issues at the societal level. I can't think of the last time I personally was mean to an obese person, despite having ample opportunities. In that context, I think the feelings of particular obese people on the issue becomes less important)

Comment author: spencerth 31 May 2010 07:18:43PM 7 points [-]

Very good article. One thing I'd like to see covered are conditions that are "treatable" with good lifestyle choices, but whose burden is so onerous that no one would consider them acceptable. Let's say you have a genetic condition which causes you to gain much more weight (5x, 10x - the number is up to the reader) than a comparable non-affected person. So much that the only way you can prevent yourself from becoming obese is to strenuously exercise 8 hours a day. If a person chooses not to do this, are they really making a "bad" choice? Is it still their fault? In this scenario, 1/3 of your day/life has become about treating this condition. I doubt too many people would honestly choose to do the "virtuous" thing in this situation.

Second thing I'd like covered: things that were inflicted on you without your consent. How much blame can you take for, let's say, your poor job prospects if your parents beat you severely every day (giving you slight brain damage of some kind, but not enough for it to be casually noticeable), fed you dog food and dirt sandwiches until you were 18, or forced you to live in an area where bullets flew into your room while you slept, forcing you to wake up in terror? There's plenty of evidence for the potentially devastating and permanent effects of trauma, poor childhood nutrition, and stress. Sure, some people manage to live like that and come out of it OK, but can everyone? Is it still right to hold someone so treated /morally/ responsible for doing poorly in their life?

Comment author: Yvain 31 May 2010 09:27:20PM 8 points [-]

If there's some cure for the genetic condition, naturally I'd support that. Otherwise, I think it would fall under the category of "the cost of the blame is higher than the benefits would be." It's not part of this person's, or my, or society's, or anyone's preferences that this person exercise eight hours a day to keep up ideal weight, so there's no benefit to blaming them until they do.

As for the second example, regarding "is it still right to hold someone so treated /morally/ responsible for doing poorly in their life", this post could be summarized as "there's no such thing as moral responsibility as a primitive object". These people aren't responsible if they're poor, just like a person with a wonderful childhood isn't responsible if they're poor, but if we have evidence that holding them responsible helps them build a better life, we might as well treat them as responsible anyway.

(the difference, I think, is that we have much more incentive to help the person with the terrible childhood, because one could imagine that this person would respond well to help; the person with the great childhood has already had a lot of help and we have no reason to think that giving more will be of any benefit)

Comment author: gjm 30 May 2010 10:48:05PM 2 points [-]

Yvain, you have a couple of instances of "(LINK)" in your text. I expect you intended to replace them with links :-).

Comment author: Yvain 30 May 2010 10:56:09PM 1 point [-]

I can't imagine what would possibly have given you that idea. (@$!%. Fixed.)

Diseased thinking: dissolving questions about disease

236 Post author: Yvain 30 May 2010 09:16PM

Related to: Disguised Queries, Words as Hidden Inferences, Dissolving the Question, Eight Short Studies on Excuses

Today's therapeutic ethos, which celebrates curing and disparages judging, expresses the liberal disposition to assume that crime and other problematic behaviors reflect social or biological causation. While this absolves the individual of responsibility, it also strips the individual of personhood, and moral dignity

             -- George Will, townhall.com

Sandy is a morbidly obese woman looking for advice.

Her husband has no sympathy for her, and tells her she obviously needs to stop eating like a pig, and would it kill her to go to the gym once in a while?

Her doctor tells her that obesity is primarily genetic, and recommends the diet pill orlistat and a consultation with a surgeon about gastric bypass.

Her sister tells her that obesity is a perfectly valid lifestyle choice, and that fat-ism, equivalent to racism, is society's way of keeping her down.

When she tells each of her friends about the opinions of the others, things really start to heat up.

Her husband accuses her doctor and sister of absolving her of personal responsibility with feel-good platitudes that in the end will only prevent her from getting the willpower she needs to start a real diet.

Her doctor accuses her husband of ignorance of the real causes of obesity and of the most effective treatments, and accuses her sister of legitimizing a dangerous health risk that could end with Sandy in hospital or even dead.

Her sister accuses her husband of being a jerk, and her doctor of trying to medicalize her behavior in order to turn it into a "condition" that will keep her on pills for life and make lots of money for Big Pharma.

Sandy is fictional, but similar conversations happen every day, not only about obesity but about a host of other marginal conditions that some consider character flaws, others diseases, and still others normal variation in the human condition. Attention deficit disorder, internet addiction, social anxiety disorder (as one skeptic said, didn't we used to call this "shyness"?), alcoholism, chronic fatigue, oppositional defiant disorder ("didn't we used to call this being a teenager?"), compulsive gambling, homosexuality, Aspergers' syndrome, antisocial personality, even depression have all been placed in two or more of these categories by different people.

Sandy's sister may have a point, but this post will concentrate on the debate between her husband and her doctor, with the understanding that the same techniques will apply to evaluating her sister's opinion. The disagreement between Sandy's husband and doctor centers around the idea of "disease". If obesity, depression, alcoholism, and the like are diseases, most people default to the doctor's point of view; if they are not diseases, they tend to agree with the husband.

The debate over such marginal conditions is in many ways a debate over whether or not they are "real" diseases. The usual surface level arguments trotted out in favor of or against the proposition are generally inconclusive, but this post will apply a host of techniques previously discussed on Less Wrong to illuminate the issue.

What is Disease?

In Disguised Queries , Eliezer demonstrates how a word refers to a cluster of objects related upon multiple axes. For example, in a company that sorts red smooth translucent cubes full of vanadium from blue furry opaque eggs full of palladium, you might invent the word "rube" to designate the red cubes, and another "blegg", to designate the blue eggs. Both words are useful because they "carve reality at the joints" - they refer to two completely separate classes of things which it's practically useful to keep in separate categories. Calling something a "blegg" is a quick and easy way to describe its color, shape, opacity, texture, and chemical composition. It may be that the odd blegg might be purple rather than blue, but in general the characteristics of a blegg remain sufficiently correlated that "blegg" is a useful word. If they weren't so correlated - if blue objects were equally likely to be palladium-containing-cubes as vanadium-containing-eggs, then the word "blegg" would be a waste of breath; the characteristics of the object would remain just as mysterious to your partner after you said "blegg" as they were before.

"Disease", like "blegg", suggests that certain characteristics always come together. A rough sketch of some of the characteristics we expect in a disease might include:

1. Something caused by the sorts of thing you study in biology: proteins, bacteria, ions, viruses, genes.
2. Something involuntary and completely immune to the operations of free will
3. Something rare; the vast majority of people don't have it
4. Something unpleasant; when you have it, you want to get rid of it
5. Something discrete; a graph would show two widely separate populations, one with the disease and one without, and not a normal distribution.
6. Something commonly treated with science-y interventions like chemicals and radiation.

Cancer satisfies every one of these criteria, and so we have no qualms whatsoever about classifying it as a disease. It's a type specimen, the sparrow as opposed to the ostrich. The same is true of heart attack, the flu, diabetes, and many more.

Some conditions satisfy a few of the criteria, but not others. Dwarfism seems to fail (5), and it might get its status as a disease only after studies show that the supposed dwarf falls way out of normal human height variation. Despite the best efforts of transhumanists, it's hard to convince people that aging is a disease, partly because it fails (3). Calling homosexuality a disease is a poor choice for many reasons, but one of them is certainly (4): it's not necessarily unpleasant.

The marginal conditions mentioned above are also in this category. Obesity arguably sort-of-satisfies criteria (1), (4), and (6), but it would be pretty hard to make a case for (2), (3), and (5).

So, is obesity really a disease? Well, is Pluto really a planet? Once we state that obesity satisfies some of the criteria but not others, it is meaningless to talk about an additional fact of whether it "really deserves to be a disease" or not.

If it weren't for those pesky hidden inferences...

Hidden Inferences From Disease Concept

The state of the disease node, meaningless in itself, is used to predict several other nodes with non-empirical content. In English: we make value decisions based on whether we call something a "disease" or not.

If something is a real disease, the patient deserves our sympathy and support; for example, cancer sufferers must universally be described as "brave". If it is not a real disease, people are more likely to get our condemnation; for example Sandy's husband who calls her a "pig" for her inability to control her eating habits. The difference between "shyness" and "social anxiety disorder" is that people with the first get called "weird" and told to man up, and people with the second get special privileges and the sympathy of those around them.

And if something is a real disease, it is socially acceptable (maybe even mandated) to seek medical treatment for it. If it's not a disease, medical treatment gets derided as a "quick fix" or an "abdication of personal responsibility". I have talked to several doctors who are uncomfortable suggesting gastric bypass surgery, even in people for whom it is medically indicated, because they believe it is morally wrong to turn to medicine to solve a character issue.

                   Graph of concept of "disease"

While a condition's status as a "real disease" ought to be meaningless as a "hanging node" after the status of all other nodes have been determined, it has acquired political and philosophical implications because of its role in determining whether patients receive sympathy and whether they are permitted to seek medical treatment.

If we can determine whether a person should get sympathy, and whether they should be allowed to seek medical treatment, independently of the central node "disease" or of the criteria that feed into it, we will have successfully unasked the question "are these marginal conditions real diseases" and cleared up the confusion.

Sympathy or Condemnation?

Our attitudes toward people with marginal conditions mainly reflect a deontologist libertarian (libertarian as in "free will", not as in "against government") model of blame. In this concept, people make decisions using their free will, a spiritual entity operating free from biology or circumstance. People who make good decisions are intrinsically good people and deserve good treatment; people who make bad decisions are intrinsically bad people and deserve bad treatment. But people who make bad decisions for reasons that are outside of their free will may not be intrinsically bad people, and may therefore be absolved from deserving bad treatment. For example, if a normally peaceful person has a brain tumor that affects areas involved in fear and aggression, they go on a crazy killing spree, and then they have their brain tumor removed and become a peaceful person again, many people would be willing to accept that the killing spree does not reflect negatively on them or open them up to deserving bad treatment, since it had biological and not spiritual causes.

Under this model, deciding whether a condition is biological or spiritual becomes very important, and the rationale for worrying over whether something "is a real disease" or not is plain to see. Without figuring out this extremely difficult question, we are at risk of either blaming people for things they don't deserve, or else letting them off the hook when they commit a sin, both of which, to libertarian deontologists, would be terrible things. But determining whether marginal conditions like depression have a spiritual or biological cause is difficult, and no one knows how to do it reliably.

Determinist consequentialists can do better. We believe it's biology all the way down. Separating spiritual from biological illnesses is impossible and unnecessary. Every condition, from brain tumors to poor taste in music, is "biological" insofar as it is encoded in things like cells and proteins and follows laws based on their structure.

But determinists don't just ignore the very important differences between brain tumors and poor taste in music. Some biological phenomena, like poor taste in music, are encoded in such a way that they are extremely vulnerable to what we can call social influences: praise, condemnation, introspection, and the like. Other biological phenomena, like brain tumors, are completely immune to such influences. This allows us to develop a more useful model of blame.

The consequentialist model of blame is very different from the deontological model. Because all actions are biologically determined, none are more or less metaphysically blameworthy than others, and none can mark anyone with the metaphysical status of "bad person" and make them "deserve" bad treatment. Consequentialists don't on a primary level want anyone to be treated badly, full stop; thus is it written: "Saddam Hussein doesn't deserve so much as a stubbed toe." But if consequentialists don't believe in punishment for its own sake, they do believe in punishment for the sake of, well, consequences. Hurting bank robbers may not be a good in and of itself, but it will prevent banks from being robbed in the future. And, one might infer, although alcoholics may not deserve condemnation, societal condemnation of alcoholics makes alcoholism a less attractive option.

So here, at last, is a rule for which diseases we offer sympathy, and which we offer condemnation: if giving condemnation instead of sympathy decreases the incidence of the disease enough to be worth the hurt feelings, condemn; otherwise, sympathize. Though the rule is based on philosophy that the majority of the human race would disavow, it leads to intuitively correct consequences. Yelling at a cancer patient, shouting "How dare you allow your cells to divide in an uncontrolled manner like this; is that the way your mother raised you??!" will probably make the patient feel pretty awful, but it's not going to cure the cancer. Telling a lazy person "Get up and do some work, you worthless bum," very well might cure the laziness. The cancer is a biological condition immune to social influences; the laziness is a biological condition susceptible to social influences, so we try to socially influence the laziness and not the cancer.

The question "Do the obese deserve our sympathy or our condemnation," then, is asking whether condemnation is such a useful treatment for obesity that its utility outweights the disutility of hurting obese people's feelings. This question may have different answers depending on the particular obese person involved, the particular person doing the condemning, and the availability of other methods for treating the obesity, which brings us to...

The Ethics of Treating Marginal Conditions

If a condition is susceptible to social intervention, but an effective biological therapy for it also exists, is it okay for people to use the biological therapy instead of figuring out a social solution? My gut answer is "Of course, why wouldn't it be?", but apparently lots of people find this controversial for some reason.

In a libertarian deontological system, throwing biological solutions at spiritual problems might be disrespectful or dehumanizing, or a band-aid that doesn't affect the deeper problem. To someone who believes it's biology all the way down, this is much less of a concern.

Others complain that the existence of an easy medical solution prevents people from learning personal responsibility. But here we see the status-quo bias at work, and so can apply a preference reversal test. If people really believe learning personal responsibility is more important than being not addicted to heroin, we would expect these people to support deliberately addicting schoolchildren to heroin so they can develop personal responsibility by coming off of it. Anyone who disagrees with this somewhat shocking proposal must believe, on some level, that having people who are not addicted to heroin is more important than having people develop whatever measure of personal responsibility comes from kicking their heroin habit the old-fashioned way.

But the most convincing explanation I have read for why so many people are opposed to medical solutions for social conditions is a signaling explanation by Robin Hans...wait! no!...by Katja Grace. On her blog, she says:

...the situation reminds me of a pattern in similar cases I have noticed before. It goes like this. Some people make personal sacrifices, supposedly toward solving problems that don’t threaten them personally. They sort recycling, buy free range eggs, buy fair trade, campaign for wealth redistribution etc. Their actions are seen as virtuous. They see those who don’t join them as uncaring and immoral. A more efficient solution to the problem is suggested. It does not require personal sacrifice. People who have not previously sacrificed support it. Those who have previously sacrificed object on grounds that it is an excuse for people to get out of making the sacrifice. The supposed instrumental action, as the visible sign of caring, has become virtuous in its own right. Solving the problem effectively is an attack on the moral people.

A case in which some people eat less enjoyable foods and exercise hard to avoid becoming obese, and then campaign against a pill that makes avoiding obesity easy demonstrates some of the same principles.

There are several very reasonable objections to treating any condition with drugs, whether it be a classical disease like cancer or a marginal condition like alcoholism. The drugs can have side effects. They can be expensive. They can build dependence. They may later be found to be placebos whose efficacy was overhyped by dishonest pharmaceutical advertising.. They may raise ethical issues with children, the mentally incapacitated, and other people who cannot decide for themselves whether or not to take them. But these issues do not magically become more dangerous in conditions typically regarded as "character flaws" rather than "diseases", and the same good-enough solutions that work for cancer or heart disease will work for alcoholism and other such conditions (but see here).

I see no reason why people who want effective treatment for a condition should be denied it or stigmatized for seeking it, whether it is traditionally considered "medical" or not.

Summary

People commonly debate whether social and mental conditions are real diseases. This masquerades as a medical question, but its implications are mainly social and ethical. We use the concept of disease to decide who gets sympathy, who gets blame, and who gets treatment.

Instead of continuing the fruitless "disease" argument, we should address these questions directly. Taking a determinist consequentialist position allows us to do so more effectively. We should blame and stigmatize people for conditions where blame and stigma are the most useful methods for curing or preventing the condition, and we should allow patients to seek treatment whenever it is available and effective.

Comment author: Nick_Tarleton 26 May 2010 08:27:24PM 7 points [-]

You're an equivalence class. You don't save the last eight billion humans, you save eight billion humans in each of the infinitely many worlds in which your decision algorithm is instantiated.

Comment author: Yvain 29 May 2010 10:14:30PM 1 point [-]

Why is that significant? No matter how many worlds I'm saving eight billion humans in, there are still humans left over who are saved no matter what I do or don't do. So the "reward" of my actions still gets downgraded from "preventing human extinction" to "saving a bunch of people, but humanity will be safe no matter what".

In fact...hmm...any given human will be instantiated in infinitely many worlds, so you don't actually save any lives. You just increase those lives' measure, which is sort of hard to get excited about.

Comment author: CarlShulman 26 May 2010 09:12:42PM 8 points [-]

your subjective experience has an equal chance of continuing down any branch

And just what does that mean?

Comment author: Yvain 26 May 2010 09:57:31PM 2 points [-]

Maybe nothing - maybe the fundamental unit of conscious experience is the observer-moment and that continuity of experience is an illusion - but the consensus on this site seems to be that it's worth talking about in situations like eg quantum suicide or simulation.

Comment author: Kutta 26 May 2010 01:13:24PM *  2 points [-]

reducing the percentage of branches of humanity that end in destruction by an insignificant

Well, it definitely sounds worse than simply saving the world, but the expected number of saved lives should be the same either ways.

Comment author: Yvain 26 May 2010 04:50:49PM 4 points [-]

Yes, but utility isn't linear across saved lives and maybe it even shouldn't be. I would be willing to give many more resources to save the lives of the last fifty pandas in the world, saving pandas from extinction, than I would be to save fifty pandas if total panda population was 100,000 threatening to go down to 99,950.

Now it's true that human utility is more linear than panda utility because I care much more about humans for their own sake versus for the sake of my preference for there being humans, but I still think saving the last eight billion humans is more important than saving eight billion out of infinity.

Comment author: Roko 26 May 2010 11:36:44AM 0 points [-]

Does this theory really alter the probability that your next chocolate bar will turn into a hamster?

After all, if there were only one of you, maybe there's a one in a trillion chance that one is in a simulation whose alien overlords will turn a chocolate bar into a hamster.

But what if there are infinity of you, and the set of you that are not in simulations has measure 0? Then the probability of bizarre things happening is much higher, and depends entirely upon the probability distribution over motivations of simulators.

Comment author: Yvain 26 May 2010 04:46:04PM 0 points [-]

Isn't the measure of the set of me not in simulations (in a big world) equal to the probability that I'm not in a simulation (if there's only one of me)?

Comment author: Yvain 26 May 2010 10:30:18AM *  25 points [-]

Does this theory really alter the probability that your next chocolate bar will turn into a hamster? After all, if there were only one of you, maybe there's a one in a trillion chance that one is in a simulation whose alien overlords will turn a chocolate bar into a hamster. If there are a trillion of you, and one of those trillion is in such a simulation, and your subjective experience has an equal chance of continuing down any branch, then the probability of the bar turning into the hamster is still one in a trillion. Although I've never seen a proof, intuitively you'd expect those two probabilities to be the same, or at least not be able to predict how they differ.

It all adds up to normality...except that this takes a lot of the oomph out of the project to reduce existential risk. Saving all humanity from destruction makes a much better motivator for me than reducing the percentage of branches of humanity that end in destruction by an insignificaaEEEEGH MY KEYBOARD JUST TURNED INTO A BADGER!!11asdaghf

Comment author: SilasBarta 25 May 2010 10:10:06PM 1 point [-]

Wow! Now that you mention that article, I think I had solved the unsolved problem Eliezer describes in it, back in a discussion from a month ago, not realizing that my position on it wasn't the standard one here!

Someone tell me if I'm missing something here: Eliezer is saying that utility that a hypothesis predicts (from a course of action) can increase much faster than the length of the hypothesis. Therefore, you could feed an ideal AI a prediction that is improbable, but with a large enough utility to make it nevertheless highly important. This would force the AI to give in to Pascal's muggings.

My response (which I assumed was the consensus!) was that, when you permit a hypothesis long enough to associate that mega-utility with that course of action, you are already looking at very long hypotheses. When you allow all of those into consideration, you will necessarily allow hypotheses with similar probability but believe the opposite utility from that COA.

Because the mugger has not offered evidence to favor his/her hypothesis over the opposite, you assign, on net, no significant expected (dis)utility to what the mugger claims to do.

Comment author: Yvain 25 May 2010 10:18:55PM 12 points [-]

If a normal mugger holds up a gun and says "Give me money or I'll shoot you", we consider the alternate hypotheses that the mugger will only shoot you if you do give er the money, or that the mugger will give you millions of dollars to reward your bravery if you refuse. But the mugger's word itself, and our theory of mind on the things that tend to motivate muggers, make both of these much less likely than the garden-variety hypothesis that the mugger will shoot you if you don't give the money. Further, this holds true whether the mugger claims er weapon is a gun, a ray gun, or a black hole generator; the credibility that the mugger can pull off er threat decreases if e says e has a black hole generator, but not the general skew in favor of worse results for not giving the money.

Why does that skew go away if the mugger claims to be holding an unfriendly AI or the threat of divine judgment some other Pascal-level weapon?

Your argument only seems to hold if there is no mugger and we're considering abstract principles - ie maybe I should clap my hands on the tiny chance that it might set into effect a chain reaction that will save 3^^^3 lives. In those cases, I agree with you; but as soon as a mugger gets into the picture e provides more information and skews the utilities in favor of one action.

Comment author: Thomas 22 May 2010 08:32:44PM -1 points [-]

I know. But not long ago, nobody expected that a bacteria is to blame. On the contrary! It was postulated, that no bacteria could possibly survive the stomach environment.

Comment author: Yvain 22 May 2010 08:41:41PM *  2 points [-]

So what are you suggesting with that example? That we should pre-emptively treat all diseases with antibiotics just in case bacteria are to blame?

Comment author: Thomas 22 May 2010 05:57:03PM -1 points [-]

This is one example. Maybe as free as the aspirin antibiotics would do here:

Link

Comment author: Yvain 22 May 2010 07:47:25PM *  3 points [-]

All serious cases of stomach/duodenal ulcer are already tested for h. pylori and treated with several different antibiotics if found positive.

Comment author: Thomas 22 May 2010 02:28:24PM -2 points [-]

Antibiotics. The common wisdom is, that we use them too much. Might be, that the opposite is true. A more massive poisoning of pathogens with antibiotics could push them over the edge, to the oblivion. This way, when we use the antibiotics reluctantly, we give them a chance to adapt and to flourish.

It just might be.

Comment author: Yvain 22 May 2010 02:50:29PM 10 points [-]

Do you have a citation for that?

As far as I understand it, when giving antibiotics to a specific patient, doctors often follow your advice - they give them in overwhelming force to eradicate the bacteria completely. For example, they'll often give several different antibiotics so that bacteria that develop resistance to one are killed off by the others before they can spread. Side effects and cost limit how many antibiotics you give to one patient, but in principle people aren't deliberately scrimping on the antibiotics in an individual context.

The "give as few antibiotics as possible" rule mostly applies to giving them to as few patients as possible. If there's a patient who seems likely to get better on their own without drugs, then giving the patient antibiotics just gives the bacteria a chance to become resistant to antibiotics, and then you start getting a bunch of patients infected with multiple-drug-resistant bacteria.

The idea of eradicating entire species of bacteria is mostly a pipe dream. Unlike strains of virus that have been successfully eradicated, like smallpox, most pathogenic bacteria have huge bio-reservoirs in water or air or soil or animals or on the skin of healthy humans. So the best we can hope to do is eradicate them in individual patients.

Comment author: Yvain 21 May 2010 01:01:57PM *  2 points [-]

Interesting post, good explanation of what's keeping rationality from being more practically useful, duly upvoted, but change the spelling in the title to "tragedy".

Comment author: Academian 19 May 2010 10:29:22PM *  4 points [-]

but I'm not sure whether you're even acknowledging the existence of the fourth. ... what's the equivalent of you?

Thanks for posing the question so clearly; and roughly speaking, yes, I see no need to separate the notion of "subject" and the last phases of the sensation process in the brain. I am the phenomenon of my perceptions and behavior. What else would I call "me"?

That you ask your question with an analogy is no coincidence: what leads one to ask for a subject is precisely a misapplication of metaphor. You are mapping an event chain of length N into an event chain of length N+1, and demanding that the first chain is therefore missing something: An object is like an object, a sense is a like a transmission about it, so what's like the recipient of the transmission?

Well, no... at the end, at the visual cortex or wherever visual perception happens (as mind-state "playback" technology might help us verify), the perception is not a transmission... it's the recipient of it. And "you" are just a collage of such recipients (and subsequent phenomena if you identify with your thoughts and actions too).

Comment author: Yvain 20 May 2010 12:02:41AM 3 points [-]

I hope that this series will eventually progress to an explanation of what it means for a perception to be its own recipient or for a subject to be a collage of such perceptions. It sounds promising, and I have lowered my probability that you are secretly a p-zombie (see the paragraph here starting with "He kind of took this idea and ran with it"), but I definitely want to know more.

In response to comment by Yvain on Blame Theory
Comment author: cousin_it 19 May 2010 10:56:27PM *  3 points [-]

The Shapley value has been used on LW several times already: 1, 2. I understand it as follows: imagine a game with many players that can make "coalitions" with each other to win money from the universe, and two "coalitions" joined together can always win no less than they'd have won separately. Then the Shapley value is a way of distributing the maximum total winnings (where everyone cooperates) such that every player and every group of players get no less than they could've won for themselves by defecting (individually or as a group).

(I edited this away, but now Yvain replied to it, so I'm restoring it:) Should we reward a completely ineffectual action? Are you a deontologist?

In response to comment by cousin_it on Blame Theory
Comment author: Yvain 19 May 2010 11:22:22PM *  7 points [-]

No, but guilt is an inherently deontological concept.

Let me give an example. Actually, your example. Your Hitler voter model. Yeah, it successfully makes the person who voted for Hitler feel guilty. But it also makes the person who didn't vote for Hitler, and maybe did everything e could to stop Hitler before being locked up in a German prison, equally guilty. So it actually makes the exact mistake you're warning against - unless your single vote decides whether or not Hitler gets into power, people who votes for and against Hitler end up equally guilty (if your single vote decides it, then your present welfare is greater and you have less difference between present and perfect welfare).

Guilt is there to provide negative reinforcement for acting in an immoral way. So it's only useful if there's some more moral way you could act that it needs to reinforce you towards. Loading someone who's literally done everything e could with a huge burden of guilt is like chronic pain disorder: if the pain's not there to tell you to stop doing something painful, it's just getting in the way.

And if your brain gives you equal squirts of guilt for voting for Hitler vs. fighting Hitler, guilt fails in its purpose as a motivation not to vote for Hitler, and any AI with a morality engine built around this theory of guilt will vote Hitler if there's any reason to do so at all.

(as for Shapley, I see references to it but not a good explanation of how to derive it and why it works. Maybe that's one of those things that actually can't be explained simply and I ought to bite the bullet and try to parse the wiki article.)

In response to comment by Yvain on Blame Theory
Comment author: cousin_it 19 May 2010 10:29:13PM *  1 point [-]

I think you understood it correctly. If two persons have equal levels of ability - could make equal potential contrubutions to the brave new world - then yes, equal welfare today implies equal guilt. Playing C while everyone else plays D may look noble, but if it has no effect, do we really want to encourage it? Couldn't the first person just look around and find a better use for their time?

In response to comment by cousin_it on Blame Theory
Comment author: Yvain 19 May 2010 10:46:00PM 3 points [-]

So if it's possible to do everything exactly perfectly, to the level of a superintelligence calculating how it could most increase world utility and then performing only those actions - and still end up with guilt in a sufficiently hard-to-fix situation - why are you calling this quantity "guilt" at all? It certainly doesn't fit my concept of what guilt is supposed to mean, and judging by the end of your post it doesn't fit yours.

Why not call it "variable X", and note that variable X has no particular correlation to any currently used English term or human emotion?

Also, the Shapley Value looks really interesting, but the wikipedia article you linked to sends me into Excessive Math Panic Mode. If you wanted to explain it in a more understandable/intuitive way, that would make a great topic for an LW post.

In response to Blame Theory
Comment author: Yvain 19 May 2010 10:23:39PM *  3 points [-]

Say there's a perfect person who does everything e can to create a perfect society, and really does it well, to the limits of er ability, but no one else listens and so a perfect society is not created. In fact, everyone else is hopelessly evil and society doesn't change at all as a result of er efforts.

There's a second person who sits at home all day and watches TV. Society also doesn't change at all as a result of er efforts.

Would these people still end up with the same level of guilt, given that the difference between the perfect world welfare and their current world welfare is exactly the same? Or am I misunderstanding this post as badly as I feel like I must be?

Comment author: Academian 19 May 2010 09:37:52PM *  0 points [-]

I was hoping to pre-empt this sort of confusion in my paragraph starting with

Under Descartes' influence, the language I'm using here is somewhat suggestive of dualism in its distiction between physical phenomena and our perceptions of them, but in fact it seems that our sensations simply are physical phenomena...

The point is that if physics is right, the distinction between senses and the "objects of sensation" is really just the distinction between the physical phenomenon of sensing and the penultimate causes of such phenomena, outside our bodies.

Think of it this way: of course senses are physical phenomena... you can sense them! At some point when you were a child, being able to sense something is what it meant for it to be physical. I think that's still the right idea.

Comment author: Yvain 19 May 2010 10:01:42PM 4 points [-]

You're talking about three things - object of sensation, sense, and mental representation of sensation. I'm talking about four things - object of sensation, sense, mental representation of sensation, and subject. I think we both agree on everything about the first three and that all of them are physical and inter-convertible, but I'm not sure whether you're even acknowledging the existence of the fourth.

A friend goes to Africa and sees an elephant. He takes a photograph and sends it to you in .ZIP format via email. You download it, unzip it, and you see the image.

In this case, the elephant is the object of sensation. Your friend is the sense of vision. The photograph is the sensory representation of the elephant, the email is the optic nerve, and the unzipper is the visual cortex. The thesis of this post is that your sense of vision (within the metaphor) is equivalent to your sense of consciousness (outside the metaphor), and that's fine, but then what's the equivalent of you? (wow that metaphor came out badly)

And I know that the answer in a neurological sense is that different features of the elephant activate various forms of mental processing which result in actions like saying "Wow, great elephant photo" and the like, but that's an answer that works equally well for humans and p-zombies. The philosophical answer that explains the subjective sensation is harder to come by.

Comment author: Yvain 19 May 2010 09:40:34PM *  0 points [-]

Definitely makes some sense.

But I didn't understand what you meant in the paragraph starting with "What stops us from just saying..." What does stop us from just saying this, and how come some desires successfully result in action and others result in wishful thinking? Can you predict when wishful thinking would be more likely to occur?

On a similar note, if "the final goal of a plan is a belief", would you expect me to be indifferent between saving the world and taking a pill that caused me to believe that the world was saved, or is that confusing levels?

Comment author: Yvain 19 May 2010 09:16:39PM *  7 points [-]

To anyone who really suffers from doubt about the physical nature of consciousness (...only sometimes, and I'm not proud of it) this line of thinking is pretty exasperating. Yes, the easy problem is easy, and most responsible dualists are happy enough admitting that the mind operates through physical means detectable on an MRI and we gain information about our own mental states through physical neural function...

...none of which makes any difference to the question that non-physicalists actually worry about, the so-called hard problem. Even calling consciousness a "sense" is admitting the lack of a solution to the hard problem: senses are the modalities by which objects are perceived. So our inner states are the objects, consciousness is the modality, and then who's doing the perceiving? More consciousness? Who's perceiving that? Hofstadter's solution of the strange loop is interesting but so nontechnical as to be useless.

Calling consciousness a sense, insofar as it's not just an equivocation on the term "consciousness", passes the recursive buck. The subjective level at which the senses bottom out remains just as poorly understood as before. This is more of an argument that thinking is purely physical (and a good one). I'm hoping you'll get to what I think of as "consciousness" later on in the series.

On the other hand, thanks for that link to the voice recording made in 1860. Getting to hear the oldest accessible human sound in the world? Pretty neat.

In response to Multiple Choice
Comment author: Unnamed 17 May 2010 11:43:14PM 6 points [-]

Maybe I'm misunderstanding what kinds of cases you have in mind, but I would've guessed that the bias is in the opposite direction. People tend to play it safe, choosing inaction, avoiding the worst possible outcome, and not sticking their neck out in the way that could leave them most visibly and unambiguously wrong and open to criticism.

Could you give some specific examples of cases where people seem to be biased towards guessing?

In response to comment by Unnamed on Multiple Choice
Comment author: Yvain 18 May 2010 12:09:59PM 8 points [-]

I agree with this. For example, there was a discussion a while back on opt-in versus opt-out organ donation. Most LWers supported opt-out because the downside of making a mistake in opt-out is a dead person's preferences being accidentally violated, as opposed to the downside of making a mistake in opt-in, where someone dies because they don't get an organ.

Most people in the general population prefer opt-in, and the only reason I can think of is that they feel like bad consequences through inaction (not taking an organ that should be taken) is okay in a way that bad consequences through action (taking an organ that shouldn't be taken) is not.

Or, regarding literal multiple choice tests where the bias is the other way, see Shut Up And Guess.

Comment author: Roko 16 May 2010 05:30:16PM 5 points [-]

If you're in a social situation that is without drama, then the social subcommunications will be relatively uninteresting.

However, there are many interactions where this will not be the case, e.g. dating/romance, probably business events that involve alliance and conflicting interest.

Comment author: Yvain 16 May 2010 09:39:26PM *  2 points [-]

This mirrors my experience, but then how come other people, whose lives are generally just as boring as mine, seem to like parties?

Comment author: NancyLebovitz 15 May 2010 03:35:11PM 3 points [-]

I didn't quite say that.

There's social pressure to be extroverted in the US-- but there are a lot of unhappy introverts. There's social pressure to be introverted in other places (Great Britain?), but for all I know there are a lot of squelched-feeling, lonely extroverts.

Comment author: Yvain 16 May 2010 03:19:31PM 3 points [-]

I live in Ireland, and the social pressure in favor of extraversion is at least as great as in the United States.

Comment author: Roko 15 May 2010 04:48:49PM *  11 points [-]

just plain bored with having nothing to say but the same small talk everyone else is making.

this is indicative that you are paying attention to the topic/words of the conversation, rather than the sub-communication, which is often the interesting bit for these kinds of conversations. People who can't read social cues and sub-communication typically don't get why others find small talk so "interesting", but this is rather like a radio that sends the carrier wave rather than the signal to the loudspeaker. The topics are just there as a "carrier wave" which is then modulated to encode social signals.

Sub-communication includes agreement/disagreement (someone agrees with you to signal alliance, disagrees to signal enmity), tone of voice (tone that rises towards the end of the sentence indicates submission, tone that falls towards the end conflict/assertiveness/dominance), body language, who gets to talk most/who listens most. Once you tune your radio in, you may find such occasions more exciting.

Tuning in to the carrier wave in social situations is a common failure mode for smart people, because we feel comfortable assessing factual statements.

Comment author: Yvain 16 May 2010 03:18:31PM *  4 points [-]

So if I'm with a bunch of people from my class, and I already know who's considered "high status", and none of us have any major conflict of interest that would make us want to assess whether or not the others are allies or not, wouldn't we all just be broadcasting generic "I like you well enough and consider you pretty much an equal, except in the context of this and this and this fact which we already both know quite well" signals? Go to a party with thirty people, and unless someone's committed the faux pas of inviting my arch-enemy, I do this thirty times. If anything, this seems even less interesting than literally talking about the weather.

If you can recommend a good free source of information that explains this (or a book that's worth the money), even better.

EDIT: Yeah, what Wei Dai said.

Comment author: Yvain 16 May 2010 02:52:51PM 1 point [-]

Thanks for the careful explanations. Even I was able to follow your math, which is pretty rare. The evaluation of the present value of an asset part was very interesting. I join other people in being skeptical of the immediate usefulness of the 'when to self-improve' part, but please do make the post on the practical side.

Comment author: Yvain 15 May 2010 11:50:18AM 12 points [-]

The comments to this post and most of the other literature I've read assumes that the problem with poorly social people is that they're afraid, not sure how to carry out a conversation effectively, or make poor decisions when confronted with social dilemmas.

Anecdotally, my experience isn't like this at all. I'm pretty good, maybe even better than average, at talking to people in one-to-one conversations, at home, at cafes, on the bus, before class, and pretty much any time other than at deliberately social events. But at bars or parties, the constantly shifting conversations of dozens of people trying to all talk to each other at once at a mile a minute, about nothing in particular, in a loud and overstimulating environment completely discombobulates me, and I usually end up either ignored, unable to break into a conversation more than once every few minutes, or just plain bored with having nothing to say but the same small talk everyone else is making.

Maybe I'm atypical of non-social people, but I also give a bit of credence to the possibility that all this "not knowing how to give the right reply in a conversation" stuff is what neurotypical people imagine being bad at socializing must be like, the same way hicks sometimes deal with non-English speakers by speaking English words really loud and slowly because they can't imagine what it's like to genuinely not understand English. But I'd like to hear from other non-social people to confirm.

(I got a 23 on the test)

Comment author: Nick_Tarleton 14 May 2010 12:11:23AM *  10 points [-]

Holy crap we're broken. Good info.

This is getting deep into Dark Arts territory, and according to Predictably Irrational, the opportunity to use these powers for evil has not gone unexploited.

I'm a little worried that calling (a/i)rrational persuasive techniques "Dark Arts", even in seemingly indefensible (though, it must be said, minor) cases like this, biases us against them and will make people averse to using them even when doing so is ethical and beneficial. What do others think? Am I overreacting?

Dan Ariely, who apparently learned his teaching methodology from Professor Quirrell

FWIW, I suspect most casual readers don't know about Methods of Rationality and were confused by this.

Comment author: Yvain 14 May 2010 09:33:00PM *  0 points [-]

I like the reference to "Dark Arts", but I've removed the Quirrell part on your suggestion.

Comment author: Nanani 13 May 2010 03:02:26AM 0 points [-]

Seconded, but with a request for contrast, if possible, with human-caused mass-death such as invasion by conquering hordes. What effect do such phenomena have at the genetic level wrt cognition, as opposed to cultural or lingustic transmission?

Comment author: Yvain 14 May 2010 09:22:31PM 4 points [-]

And what about human-caused mass death selecting for specific characteristics? For example, the Cambodian purges of intellectuals or the Communist purges of successful businesspeople. Are these too tenuous a proxy for genes to cause long-term change in alleles, or did the Cambodians and Communists do long-term harm to their genetic legacy?

Comment author: mattnewport 14 May 2010 01:02:36AM *  17 points [-]

Your visual system is not evolved to be a colorimeter because that is not actually very useful for the kinds of things we use our visual system for. Thinking that your brain 'should' identify the same RGB values as the same 'colors' in a different context reflects a confusion about what invariant properties of the world the visual system represents as 'color'.

Our conscious experience of color is related to the spectral composition of light that reaches our retinas but the RGB value of a pixel is not sufficient to describe the more complex qualia we label 'colors'. If there is any 'failure' captured by this illusion it is a failure to understand what a good job the brain does of extracting useful information from the complex pattern of light that falls onto our retinas rather than a failure of the visual system. A colorimeter is a relatively simple $90 device. Matching the human visual system's performance on the inverse rendering problem is an unsolved hard AI problem.

The anchoring phenomenon which can result in poor choices in certain circumstances on the other hand does reflect a 'failure' in the sense that a generally useful heuristic may lead us to make poor judgements. I'd say it is an example of misapplying a heuristic to a problem it is ill suited for. I think comparing it to the colour constancy phenomenon is misleading and inapt.

Comment author: Yvain 14 May 2010 09:15:39PM 2 points [-]

I totally agree that those functions of the visual system are features and not bugs, but I still think the analogy to biases holds; after all, there's a strong argument that that biases can be features too. It's all a question of whether they're being used in the sorts of situations for which they were designed, or whether they're in an unusual situation where they break down. In the case of vision, that's specially designed optical illusions, but practically all modern thought is outside the original design specs for our cognitive heuristics.

Comment author: thomblake 14 May 2010 03:55:25PM 1 point [-]

You're missing citations or links for the experiments you mentioned. While most of us could figure out that Predictably Irrational is a book by Ariely and the experiment is somewhere within, you should really specify that.

Comment author: Yvain 14 May 2010 08:54:53PM 0 points [-]

I unfortunately don't have the book with me, so I can't give you page numbers. I gave a link to the Amazon page for the book, but I realize that's not optimal. There's a link to an online Scribd version of the book I can give you if you want it, but it's of questionable legality and I figured having that in the main article wasn't the sort of thing we wanted LW associated with.

Comment author: Jack 14 May 2010 01:07:43AM 5 points [-]

I really doubt that same neural mechanism is involved. Like P= 0.0005. We're dealing with totally different areas of the brain that evolved hundreds of millions of years apart. I'm not even sure I see an obvious sense in which the optical illusion corresponds to the cognitive bias.

Comment author: Yvain 14 May 2010 08:52:47PM 1 point [-]

I don't think it's mechanically the same, or that there's a value-of-magazine-subscriptions equivalent to double opponent cells in the visual cortex, but I think the two processes are conceptually solutions to the same problem.

The general problem is trying to pick out salient features from what's currently in the attention without being distracted by the macro-level problem of how what's currently in the attention differs from everything else.

So in color vision, that's something like determining what parts of a field are redder or greener than others without being distracted by the entire field being redder than usual because it's sunset. In purchasing, it's something like deciding which of two meals is better value than another without being distracted by the whole menu being more expensive than normal because it's a fancy restaurant.

Comment author: Eliezer_Yudkowsky 14 May 2010 06:04:23AM 10 points [-]

Dan Ariely, who apparently learned his teaching methodology from Professor Quirrell

Epic reversed causality much?

Heuristics and biases researchers -> Eliezer Yudkowsky -> Professor Quirrell.

Comment author: Yvain 14 May 2010 08:28:53PM *  64 points [-]

This is really too long for a comment, but I couldn't resist:

Tom Marvolo Riddle, who had lately taken to calling himself Lord Voldemort, shut his eyes. When he opened them again, the envelope was lying on his bed, just as he had hoped. He studied the papers inside with barely concealed glee. Over several pages was a very long string of incomprehensible letters and numbers. Almost shaking, he typed it into the computer, where it said "ENTER RANDOM SEED", and pressed enter. The screen filled with - could it be? - pages of comprehensible text. He had to restrain himself from giving a very un-Dark-Lordly whoop of triumph.

This was outstandingly clever even for him. The Time-Turner, the computer generating random text blocks that - with the right seed - would be coherent hypotheses, the dungeon of prisoners to serve as subjects in hastily conducted impromptu experiments. And the only stable time loop being the one where he ended up with the secrets of human behavior.

He hit PRINT, then called in Dolohov, the trustworthy one. "Take these psychology hypotheses and test them on the prisoners downstairs. If you get p < .05, no, make it .01, and if you can think of ways they'd be useful in manipulating people, take these pages of letters and numbers, copy them, stick the copy in this envelope, and put it on my desk. Otherwise, increase the ASCII value of the last character in the random seed by one, and put that on my desk. No questions, just do it. And do it in the next twenty-four hours."

That would take care of the Time-Turner. He was glad his model lacked the length limitations of the standard version: six hours wouldn't have been enough time for Dolohov to finish the tests. Those idiots at the Ministry had no idea how easy it was to extend the gas mileage of one of those things. Or maybe they did and were too squeamish to go around obtaining the secret ingredient. He smiled lovingly at the jar of kitten hearts on his desk. Those things were like the duct tape of evil magic - you could use them for anything.

Now it only remained to make sure no one else ever copied his trick. The Ministry he'd already dealt with - Merlin bless that fool Minister Bagnold and her "it's inconceivable anyone could break through our wards" policy. He'd apparated into the Ministry, put a curse on their whole stockpile of Time-Turners, and gotten out with no one the wiser. The artifacts would still function for little things like attending extra classes, but try to use them for anything...clever...and they'd do their best to scare the user into submission. Cryptic warnings not to mess with time, that kind of thing.

But his own Death Eaters were more of a problem. They'd notice if their master suddenly and inexplicably learned all the secrets of the human mind. Some of the brighter ones might start asking where he'd learned them. Some of the really bright ones might try figuring out exactly what those secrets were and how to use them themselves. He needed a distraction. Something that would turn their minds away from the Dark Lord's late nights in his study tinkering with Time Turners and computers, something that would make the secrets of the mind not even worth obtaining.

Suddenly he broke out into a grin. Make something a secret, and of course people would look for it. Make it mundane, better yet make it low status, and people wouldn't give it a second thought - that was the lesson of Muggle science. Was that on the paper he'd given Dolohov? It must be. He'd tell everyone he learned the secrets from the Muggles. Everyone knew they came up with a bright idea every so often, but the ancient and noble bloodlines who populated the Death Eaters would sooner die than go rooting around in Muggle books like Mudbloods trying to find them.

He'd fake an identity as a Muggle scientist - maybe even two, now that he'd mastered duplication with the Time-Turner, and get the secrets published in a Muggle journal. If anyone asked, he'd been reading Muggle journals, and there was nothing unusual about his possessing the secrets of the human mind at all.

His Quirrell alias was already a professor, but he was saving that for a rainy day. And Voldemort and Tom Riddle were right out. That meant someone new. He took out his sketchpad. He did so love anagrams. That TOM MARVOLO RIDDLE == I AM LORD VOLDEMORT one still sent chills up his spine. He'd want something at least that good if he was going to be a Muggle researcher or two. After a few abortive attempts, he finally found something he liked.

DOR DANIEL KM & DOR AMOS TVERSKY, IL == YIKES, I'M DARK LORD VOLDEMORT-SAN

Right, then. Dr. Daniel Kahneman and Dr. Amos Tversky, Israel. And while he was in the area, he could pick up the Spear of Destiny. Kill two birds with one stone.

He'd just need a few supplies, and he'd be ready to go. He picked up his miniature scapel and put in a call to the local animal shelter, whistling merrily to himself.

Blue- and Yellow-Tinted Choices

49 Post author: Yvain 13 May 2010 10:35PM

A man comes to the rabbi and complains about his life: "I have almost no money, my wife is a shrew, and we live in a small apartment with seven unruly kids. It's messy, it's noisy, it's smelly, and I don't want to live."
The rabbi says, "Buy a goat."
"What? I just told you there's hardly room for nine people, and it's messy as it is!"
"Look, you came for advice, so I'm giving you advice. Buy a goat and come back in a month."
In a month the man comes back and he is even more depressed: "It's gotten worse! The filthy goat breaks everything, and it stinks and makes more noise than my wife and seven kids! What should I do?"
The rabbi says, "Sell the goat."
A few days later the man returns to the rabbi, beaming with happiness: "Life is wonderful! We enjoy every minute of it now that there's no goat - only the nine of us. The kids are well-behaved, the wife is agreeable - and we even have some money!"

 

-- traditional Jewish joke

 

Related to: Anchoring and Adjustment

 

Biases are “cognitive illusions” that work on the same principle as optical illusions, and a knowledge of the latter can be profitably applied to the former. Take, for example, these two cubes (source: Lotto Lab, via Boing Boing):

 

Colored cube illusion

 

The “blue” tiles on the top face of the left cube are the same color as the “yellow” tiles on the top face of the right cube; if you're skeptical you can prove it with the eyedropper tool in Photoshop (in which both shades come out a rather ugly gray).

 

The illusion works because visual perception is relative. Outdoor light on a sunny day can be ten thousand times greater than a fluorescently lit indoor room. As one psychology book put it: for a student reading this book outside, the black print will be objectively lighter than the white space will be for a student reading the book inside. Nevertheless, both students will perceive the white space as subjectively white and the black space as subjectively black, because the visual system returns to consciousness information about relative rather than absolute lightness. In the two cubes, the visual system takes the yellow or blue tint as a given and outputs to consciousness the colors of each pixel compared to that background.

 

So this optical illusion occurs when the brain judges quantities relative to their surroundings rather than based on some objective standard. What's the corresponding cognitive illusion?

 

In Predictably Irrational (relatively recommended, even though the latter chapters sort of fail to live up to the ones mentioned here) Dan Ariely asks his students to evaluate (appropriately) three subscription plans to the Economist:

 

Economist subscription table

 

Ariely asked his subjects which plan they'd buy if they needed an Economist subscription. 84% wanted the combo plan, 16% wanted the web only plan, and no one wanted the print only plan. After all, the print plan cost exactly the same as the print + web plan, but the print + web plan was obviously better. Which raises the question: why even include a print-only plan? Isn't it something of a waste of space?

 

Actually, including the print-only plan turns out to be a very good business move for the Economist. Ariely removed the print-only plan from the choices. Now the options looked like this.

 

Economist subscription table, one option removed

 

There shouldn't be any difference. After all, he'd only removed the plan no one chose, the plan no sane person would choose.

 

This time, 68% of students chose the web only plan and 32% the combo plan. That's a 52% shift in preferences between the exact same options.

 

The rational way to make the decision is to compare the value of a print subscription to the Economist (as measured by the opportunity cost of that money) to the difference in cost between the web and combo subscriptions. But this would return the same answer in both of the above cases, so the students weren't doing it that way.

 

What it looks like the students were doing was perceiving relative value in the same way the eye perceives relative color. The ugly gray of the cube appeared blue when it was next to something yellow, and yellow when it was next to something blue. In the same way, the $125 cost of the combo subscription looks like good value next to a worse deal, and bad value next to a better deal.

 

When the $125 combo subscription was placed next to a $125 plan with fewer features (print only instead of print plus web) it looked like a very good deal – the equivalent of placing an ugly gray square next to something yellow to make it look blue. Take away the yellow, or the artificially bad deal, and it doesn't look nearly as attractive.

 

This is getting deep into Dark Arts territory, and according to Predictably Irrational, the opportunity to use these powers for evil has not gone unexploited. Retailers will deliberately include in their selection a super deluxe luxury model much fancier and more expensive than they expect anyone to ever want. The theory is that consumers are balancing a natural hedonism that tells them to get the best model possible against a commitment to financial prudence. So most consumers, however much they like television, will have enough good sense to avoid buying a $2000 TV. But if the retailer carries a $4000 super-TV, the $2000 TV suddenly doesn't look quite so bad.

 

The obvious next question is “How do I use this knowledge to trick hot girls or guys into going out with me?” Dan Ariely decided to run some experiments on his undergraduate class. He took photographs of sixty students, then asked other students to rate their attractiveness. Next, he grouped the photos into pairs of equally attractive students. And next, he went to Photoshop and made a slightly less attractive version of each student: a blemish here, an asymmetry there.

 

Finally, he went around campus, finding students and showing them three photographs and asking which person the student would like to go on a date with. Two of the photographs were from one pair of photos ranked equally attractive. The third was a version of one of the two, altered to make it less attractive. So, for example, he might have two people, Alice and Brenda, who had been ranked equally attractive, plus a Photoshopped ugly version of Brenda.

 

The students overwhelmingly (75%) chose the person with the ugly double (Brenda in the example above), even though the two non-Photoshopped faces were equally attractive. Ariely then went so far as to recommend in his book that for best effect, you should go to bars and clubs with a wingman who is similar to you but less attractive. Going with a random ugly person would accomplish nothing, but going with someone similar to but less attractive than you would put you into a reference class and then bump you up to the top of the reference class, just like in the previous face experiment.

 

Ariely puts these studies in a separate chapter from his studies on anchoring and adjustment (which are also very good) but it all seems like the same process to me: being more interested in the difference between two values than in the absolute magnitude of them. All that makes anchoring and adjustment so interesting is that the two values have nothing in common with one another.

 

This process also has applications to happiness set points, status seeking, morality, dieting, larger-scale purchasing behavior, and akrasia which deserve a separate post

Comment author: SilasBarta 11 May 2010 04:06:30PM *  3 points [-]

Yes, and imagine if spammers went through the effort to make an android indistinguishable from a human on the outside (in behavior and form), and had it "spam" you after reading your internet postings/websites, on the pretense that it has some questions and wants to collaborate with you.

Then, it fakes an entire friendship, in which it gives you many useful ideas, in order to be able to slip in a few remarks here and there of the form, "Hey, I know a good Mexican pharmacy where you can get cheap Viagra." (Which you point out to your "friend" is probably a scam.)

If that's what spam comes to look like one day, I don't want a filtered inbox!

Comment author: Yvain 11 May 2010 06:27:28PM 6 points [-]

If that's what spam comes to look like one day, I don't want a filtered inbox!

http://www.smbc-comics.com/index.php?db=comics&id=1024#comic

Comment author: PeerInfinity 07 May 2010 09:48:18PM 1 point [-]

I added you as a friend on LJ.

heh, now I'm going to have to write something in the journal about what I actually think of as the conditions for qualifying as an "actual friend"... but I guess I won't try posting any more about that to this comment until I know what I actually want to say.

And I guess I might as well repeat the other warnings about the journal. I write about literally everything that seems even remotely worth writing about, and that's lots of stuff, and most of it is boring. The journal contains X-rated content, and often TMI. And then there's all the quantifiedself data, and the confusing system of tags and abbreviations...

Anyway, I guess you can see for yourself. Feedback is welcome.

random trivia: I prefer not to follow society's annoying rules for being socially obligated to exchange gifts at specific times of the year. If anyone wants to something nice for me, please just donate to SIAI instead, or possibly some other charity of your choice.

Comment author: Yvain 07 May 2010 11:34:49PM *  1 point [-]

Thanks, Peer.

Note to anyone else considering this: He is not kidding about it being huge, daunting, and unformatted. Not even a little.

Comment author: PeerInfinity 20 April 2010 07:04:10PM 1 point [-]

I checked out your recent LiveJournal posts. You seem like an interesting person too, someone who I would like as a friend.

I went ahead and added you as a friend on LJ. I guess I should warn you that the journal is full of gratuitous self-disclosure. I write about literally anything that I feel like writing about. And the quantifiedself experiment means that I document literally everything I do, though I still don't have much of a life, so this isn't all that much.

Though I guess there's no need for me to be so paranoid with these warnings, and no need for me to be so paranoid about who I give access to.

Comment author: Yvain 07 May 2010 09:03:18PM 1 point [-]

Just saw this; I'm interested in seeing the journal. My LJ username is squid314. I wouldn't be an "actual friend" as in buy you stuff for your birthday, but I check my friends page every so often and respond to anything I find interesting.

Comment author: NancyLebovitz 07 May 2010 07:27:49PM 1 point [-]

A different angle on alcoholism

The End of My Addiction is by a cardiologist who became an alcoholic and eventually found that Baclofen, an muscle relaxant, eliminated his craving for alcohol and surprisingly, his compulsive shopping.

His theory is that alcoholism is an effort to feel normal-- taking away the pleasure from the alcohol isn't a good solution.

Comment author: Yvain 07 May 2010 08:00:41PM 0 points [-]

I've heard good things about baclofen too, but I haven't looked at in in depth. If you've read that book, I'd be really interested in seeing a post about the science behind it.

Comment author: clay 07 May 2010 05:52:32AM *  1 point [-]

Could you help end news/TV enjoyment with this method? Take and then watch TV, play video games, browse reddit/CNN/etc. End political identity biases? Take and then go to a political party meeting, religious service or read a bunch of politically charged information. Food cravings? Take and stuff your face with ice cream and greasy mexican food. Dirty gossip? Take and gossip about who your friends have slept with, who hates who, etc.

Do any of these activites involve the endorphine mechanism related to drug use that it's used for currently? Are you suggesting that this is a possible ticket to becoming a LessWrong superhero?

Comment author: Yvain 07 May 2010 05:36:17PM *  1 point [-]

I don't know. There's no certain reason it shouldn't work, but I could see a couple of possible sticking points.

Alcoholism's a much more simple chemical addiction than some of the others. I don't know if those are mediated by opioids, dopamine, or something else entirely (though it would be worth study). And if it works, without some countermethod like the one Roko's hinting at in his comment, you're going to kill your ability to enjoy everything you currently enjoy without gaining any new enjoyment of anything else, which sounds as likely to leave you depressed as superproductive.

Oh, and the alcoholics have to take the drug before each time they drink alcohol, even if it's twenty years after the finish the original course and they've been sober the whole time. Otherwise they get the reinforcement and the alcoholism starts to return. So I doubt anyone could practically arrange to take a tablet a half hour before each time they think about in-group politics.

In a few years I might be in a better position to actually do some experiments with this stuff, at which point I'll report back.

Comment author: komponisto 06 May 2010 06:17:52PM *  4 points [-]

My question for Less Wrong: Just how innocent is Cameron Todd Willingham? Intuitively, it seems to me that the evidence for Willingham's innocence is of higher magnitude than the evidence for Amanda Knox's innocence.

In both instances, the prosecution case amounts to roughly zero bits of evidence. However, demographics give Willingham a higher prior of guilt than Knox, perhaps by something like an order of magnitude (1 to 4 bits). I am therefore about an order of magnitude more confident in Knox's innocence than Willingham's.

Challenge question: What does an idealized form of Bayesian Justice look like?

Bayesian jurors (preferably along with Bayesian prosecutors and judges); that's really all it comes down to.

In particular, discussions about the structure of the judicial system are pretty much beside the point, in my view. (The Knox case is not about the Italian justice system, pace just about everyone.) Such systematic rules exist mostly as an attempt at correcting for predictable Bayesian failures on the part of the people involved. In fact, most legal rules of evidence are nothing but crude analogues of a corresponding Bayesian principle. For example, the "presumption of innocence" is a direct counterpart of the Bayesian prohibition against privileging the hypothesis.

There is this notion that Bayesian and legal reasoning are in some kind of constant conflict or tension, and oh-whatever-are-we-to-do as rationalists when judging a criminal case. (See here for a classic example of this kind of hand-wringing.) I would like to dispel this notion. It's really quite simple: "beyond a reasonable doubt" just means P(guilty|evidence) has to be above some threshold, like 99%, or something. In which case, if it's 85%, you don't convict. That's all there is to it. (In particular, away with this nonsense about how P(guilty|evidence) is not the quantity jurors should be interested in; of course it is!)

From our perspective as rationality-advocates, the best means of improving justice is not some systematic reform of legal systems, but rather is simply to raise the sanity waterline of the population in general.

Comment author: Yvain 06 May 2010 07:45:01PM *  3 points [-]

Now that you mention it directly, it's flabbergasting that no one's ever said what percentage level "beyond a reasonable doubt" corresponds to (legal eagles: correct me if I'm wrong). That's a pretty gaping huge deviation from a properly Bayesian legal system right there.

Antagonizing Opioid Receptors for (Prevention of) Fun and Profit

35 Post author: Yvain 05 May 2010 02:40PM

Related to: Ugh Fields, Are Wireheads Happy?

In his post Ugh Fields, Roko discussed "temporal difference learning", the process by which the brain propagates positive or negative feedback to the closest cause it can find for the feedback. For example, if he forgets to pay his bills and gets in trouble, the trouble (negative feedback) propagates back to thoughts about bills. Next time he gets a bill, he might paradoxically have even more trouble paying it, because it's become associated with trouble and negative emotions, and his brain tends to unconsciously flinch away from it.

He links to the associated Wikipedia article:

The TD algorithm has also received attention in the field of neuroscience. Researchers discovered that the firing rate of dopamine neurons in the ventral tegmental area (VTA) and substantia nigra (SNc) appear to mimic the error function in the algorithm. The error function reports back the difference between the estimated reward at any given state or time step and the actual reward received. The larger the error function, the larger the difference between the expected and actual reward. When this is paired with a stimulus that accurately reflects a future reward, the error can be used to associate the stimulus with the future reward.

Dopamine cells appear to behave in a similar manner. In one experiment measurements of dopamine cells were made while training a monkey to associate a stimulus with the reward of juice. Initially the dopamine cells increased firing rates when exposed to the juice, indicating a difference in expected and actual rewards. Over time this increase in firing back propagated to the earliest reliable stimulus for the reward. Once the monkey was fully trained, there was no increase in firing rate upon presentation of the predicted reward. This mimics closely how the error function in TD is used for reinforcement learning.

So if I understand this right, the monkey hears a bell and is unimpressed, having no expectation of reward. Then the monkey gets some juice that tastes really good and activates (opioid dependent?) reward pathways. The dopamine system is pretty surprised, and broadcasts that surprise back to all the neurons that have been especially active recently, most notably the neurons that activated upon hearing the bell. These neurons are now more heavily associated with the dopamine system. So the next time the monkey hears a bell, it has a greater expectation of reward.

And in this case it doesn't matter, because the monkey can't do anything about it. But if it were a circus monkey, and its trainer was trying to teach it to do a backflip to get juice, the association between backflips and juice would be pretty useful. As long as the monkey wanted juice, merely entertaining the plan of doing a backflip would have motivational value that promotes the correct action.

The Sinclair Method is a promising technique for treating alcoholics that elegantly demonstrates these pathways by sabotaging them.

Alcohol produces a surge of opioids in, yes, the ventral tegmental area. The temporal difference algorithm there correctly deduces that the reward is due to alcohol, and so links the dopamine system to things like drinking, planning to drink, et cetera. Rounding the nearest cliche, dopamine represents "wanting", so this makes people want to drink.

Repeat this process enough, or start with the right (wrong?) chemical structure for your opioid and dopamine receptors, and you become an alcoholic.

So to treat alcoholism, all you should have to do is reverse the process. Drink something, but have it not activate the reward system at all. Those dopaminergic neurons that detect error in your reward predictions start firing like mad and withdrawing their connections to the parts of the brain representing drinking, drinking is no longer associated with "wanting", you don't want to drink, and suddenly you're not an alcoholic any more.

It's not quite that easy. But it might be pretty close.

The Sinclair Method of treating alcoholism is to give patients naltrexone, an opioid antagonist. Then the patients are told they can drink as much as they want. Then they do. Then they gradually stop craving drink.

In these people, alcohol still produces opioids, but the naltrexone prevents them from working and they don't register with the brain's reward system. Drinking isn't "fun" any more. The dopamine system notices there's no reward, and downgrades the connection between reward and drinking, which from the inside feels like a lessened craving to drink.

In theory, this same process should be useful against any addiction or unwanted behavior. In practice, research either supports or is still investigating naltrexone use1 against smoking, self-harm, kleptomania, and overeating (no word yet on Reddit use).

The method boasts an success rate of between 25% to 78% on alcoholics depending on how you define success. A lot of alcoholism statistics are comparing apples to oranges (did they stay sober for more than a year? Forever? If they just lapsed once or twice, does that still count?) but eyeballing the data2 makes this look significantly better than either Alcoholics Anonymous or willpower alone.

I'm kind of confused by the whole idea because I don't understand the lack of side effects. Knocking out the brain's learning system to cure alcoholism seems disproportionate, and I would also expect naltrexone to interfere with the ability to experience happiness (which many people seem to like). But I haven't heard anyone mention any side-effects along the lines of "oh, and people on this drug can never learn anything or have fun ever again", and you'd think somebody would have noticed. If anyone on Less Wrong has ever used this method, or used naltrexone for anything else, please speak up.

Since these same pathways control so many cravings besides alcoholism, research in this area will probably uncover more knowledge of what really motivates us.

Footnotes

1: There's a subtle but important difference between the Sinclair Method and simple naltrexone use. As I understand it, most doctors who prescribe naltrexone tell the patient to abstain from alcohol as much as possible, but the Sinclair Method tells the patients to continue drinking normally. There are also some complicated parts about exactly when and how often you take the drug. The theory predicts the Sinclair Method would have better results, and the data seems to bear this out. As far as I know, all the studies on kleptomania, overeating, et cetera have been done on standard naltrexone use, not the Sinclair Method; I predict the Sinclair Method would work better, although there might be some practical difficulties invovled in telling a kleptomaniac "Okay, take this tablet once a day while stealing stuff at the same rate you usually do."

2: 27% "never relapse into heavy drinking" and 78% get drinking "below the level of increased risk of morbidity and mortality". There's also an 87% number floating around without any justification or link to a study. I think this guy's statistics on a ~5-10% yearly remission rate from willpower or AA sound plausible.

 

Comment author: Alicorn 04 May 2010 08:05:46PM *  5 points [-]

One that I didn't want to include in the post because I felt it would make it too inflammatory is this reaction to a particular conspiracy theory.

If anyone's read the book "Matilda" (yes, yes, fictional evidence - I remark on plausibility only), they may remember the chillingly feasible technique of the abusive headmistress to pull stunts so outrageous that the students can't get their parents to believe them. Surely someone would have noticed if the principal of a school had picked up a girl by her pigtails and flung her.

The heuristic of dismissing things that it seems someone would have noticed probably usually works, but the things that it wouldn't work on are really big, and so I'm wary of it.

Comment author: Yvain 04 May 2010 11:25:35PM *  3 points [-]

It only fails in cases where you wouldn't notice if somebody else had noticed. In a school full of terrified children, each of whom incurs a huge risk in speaking up unilaterally / going to the media about the evil headmistress, it's easy to believe that no one would have said anything. If it happened today, in the real world, I'd check www.ratemyteachers.com, where the incentives to rat on the headmistress are totally different.

The dominating principle (pun totally intended) is:

P(you heard about someone noticing|it's true) = P(you would have heard someone noticed|someone noticed) * P(someone noticed|it's true)

From there you can subtract from one to find the probability that you haven't heard about anyone noticing given that it's true, and then use Bayes' Rule to find the chance that it's true, given that you haven't heard about anyone noticing...

...I think; I don't trust my brain with any math problem longer than two steps, and I probably wrote several of those probabilities wrong. But the point is, you can do math to it, and the higher the probability that someone would have noticed if it wasn't true, and the higher the probability that you would have heard about it if someone noticed, the higher the probability that, given you haven't heard of anyone noticing it's true, it's not true.

For you to justify the rule in this post, you'd have to prove that people either systematically overestimate the chance that they'd hear of it if someone noticed, or the probability that someone would notice it if it were true.

Comment author: NancyLebovitz 01 May 2010 04:03:31PM 0 points [-]

As I understand it, Gurdieff and such are claiming it's possible to have sufficiently reliable knowledge such that basing action on anything else obviously isn't attractive.

That kind of certainty does exist in some realms-- if someone claims to have trisected the angle or built a perpetual motion machine, you can be sure there's a mistake or fraud somewhere, and you also aren't going to spend your time trying to achieve those projects yourself.

Whether such knowledge is possible for more complex situations isn't obvious, but I do think that's where he's pointing.

Comment author: Yvain 01 May 2010 04:35:56PM *  4 points [-]

Reading the quote and your explanation, I thought of this:

Through my mind flashed the passage:

"Do nothing because it is righteous, or praiseworthy, or noble, to do so; do nothing because it seems good to do so; do only that which you must do, and which you cannot do in any other way."

Doing what it seemed good to do, had only led me astray.

So I called a full stop.

And I decided that, from then on, I would follow the strategy that could have saved me if I had followed it years ago: Hold my FAI designs to the higher standard of not doing that which seemed like a good idea, but only that which I understood on a sufficiently deep level to see that I could not do it in any other way.

-- My Bayesian Enlightenment

Comment author: Morendil 21 April 2010 06:13:10PM 0 points [-]

Yvain, are you going to follow up on this now that you seem to have somewhat more time for participation here? ;)

Comment author: Yvain 28 April 2010 01:51:05PM 2 points [-]

Short answer: I had just read an article on a book called "The Root of Thought" which made it sound like it was making a very convincing case for a lot of higher thought being based in glial cells and not neurons.

It would have been fun and educational to get everyone to say they were 99.999% confident that thought was neural (which I would have done before reading the summary) and then spring the whole glial cell thing on them.

But I ended up not having time to read or even acquire the book, and no one really took the bait anyway. But yeah, "Root of Thought". If any of you have read it, please tell me what you think.

Comment author: Alicorn 27 April 2010 05:01:13PM 3 points [-]

Why is "claim an objective morality" on the list of things you shouldn't post against consensus about? I'm a moral realist; historically this has gotten me only slightly heckled, not decried as an obvious amateur.

Comment author: Yvain 28 April 2010 12:17:45PM 3 points [-]

How about "claim a universally compelling morality"?

Comment author: Yvain 27 April 2010 12:58:45PM *  3 points [-]

Everyone who says this site is intimidating to new people is spot-on. I'd read all the OB sequences by the time LW started up, but I still came very close to never writing any top-level posts, and if my first one hadn't been well received I doubt I ever would have written a second.

But the current options don't address how, as Matt Simpson said, looking dumb is a bigger barrier than karma. I got over my "looking dumb" fear by using a name that's pretty hard to trace back to me and maintaining anonymity, but this isn't enough. Internet etiquette generally frowns on re-registering with a new handle after you embarrass your first one, so I was still risking "my" Less Wrong reputation.

So what about short "karma holidays" every so often? During a karma holiday, people have the option to post/comment anonymously (wait! hear me out!). The karma they earn or lose doesn't get added or subtracted to their real total. At any point, the poster can then "claim" the post and remove anonymity from it.

During a karma holiday, new people get every incentive to post and comment. If their post bombs, they don't lose any karma, no one ever has to know it was them, and they know not to post any more until they've done some more study. If their post succeeds, they can take the karma and the credit, lose their feelings of unworthiness, and then keep on posting good stuff after the karma holiday's over.

The downside is that we have to suffer through some posts that might be lower quality than normal once in a while, and we might get flooded with too many posts to read and review properly during that period.

Comment author: Kevin 21 April 2010 11:52:54AM 3 points [-]
Comment author: Yvain 21 April 2010 06:04:34PM *  12 points [-]

pw0ncakes writes:

The lower you go on the academic totem pole, the less tolerance there is. Extensions are hard to get at state schools-- even the really good ones-- and practically nonexistent at community colleges. It may be unfair, but it makes sense. Teacher:student ratios are lower at elite colleges, and late work is rarer, so it's not really a burden. Also, professors at elite colleges are generally happier and have cushier lives, whereas teachers at CCs are usually underpaid and overworked already, so late work is an intolerable addition. Finally, many of the students in middling schools shouldn't be in college at all, while the assumption at an elite college is that the person who's late is a good student with a good reason. That assumption's not always true, of course, but elite colleges would rather make one type of error.

I really like this, because it draws some testable predictions and then tests them. The more annoying it would be for the professor to grade late work, and the more it would mess up the student's life, the less likely the professor is to accept excuses. That means professors really are making that sort of "my utility versus the student's utility" decision when deciding whether or not to accept excuses.

(although the commenter who responded that Ivy League professors do it because they don't want rich parents complaining has a point too)

Comment author: arundelo 20 April 2010 11:21:09PM 9 points [-]

Typos:

These paragraphs

You say "Like all economics professors [....] I can't grant you an extension.

A third student comes to you [....] the one whose mother died.

should end in quotation marks.

In this sentence

"The student who was here before me, that's different.

"me" should be "you".

This paragraph

You respond "Although my policies [....] sufficient justification for almost anyone breaking any law.

should end in a (double) quotation mark and most of the quotation marks in it are nested so should be single.

(Upvoted, by the way.)

Comment author: Yvain 21 April 2010 05:51:28PM 7 points [-]

You missed the "they enjoys" one. You're fired as official Less Wrong proofreader. (just kidding, thanks for the corrections)

Comment author: sketerpot 21 April 2010 03:32:40AM *  12 points [-]

You know, most of these problems can be avoided if you accept assignments late and have most of the course grade depend on one or more large individual projects that must be turned in "sometime before the end of the semester". The simplifying assumption is that a student who doesn't try to learn anything deserves to remain ignorant -- it's nice how neatly this works out.

I give my students a recommended due date, but allow them to turn in any assignment at any time before Finals Week without penalty, for any reason. It works better than you might think.

(I wish that I was allowed to insist that everybody do an individual project. There is so much cheating it's a bit ridiculous, and the worst part is that the students who cheat are just good enough at covering their tracks that I can't find absolutely definitive evidence. This is especially common among Chinese students, for some reason. Again, I figure that if people harm their own learning by cheating on assignments, then their actions are self-punishing.)

Comment author: Yvain 21 April 2010 05:50:33PM 15 points [-]

In "Predictably Irrational", they mention a study (don't have the book with me, can't be more specific) where a teacher assigns three projects, due at various points throughout the year. The students do relatively well.

In another class the same teacher assigns three projects which can be turned in whenever the students want. The students in this class do quite a bit worse than the first students because they're procrastinating and do all three projects the last week.

In yet another condition, students are allowed to turn in projects at any time during the year, but they're also allowed to voluntarily pre-commit to a due date at the beginning of the year (they fail the project if they don't have it in by their own due date). In this condition, the students who pre-committed did as well as the students in the first class, and the students who didn't pre-commit did as poorly as the students in the second class.

Based on that study, I predict your students would do better if you gave them assigned due dates.

Comment author: Jack 21 April 2010 01:39:32AM 20 points [-]

The Music Fan

A month before the paper was assigned a student was talking with their professor and mentioned that the punk band "The Exemplars" was their favorite band and that they would do anything to see them live but that they almost never perform. The day after the paper was due the student comes in and says "I'm sorry I didn't get the paper in on time, professor, but you won't believe what happened. The Exemplars, that band I told you about held a one-time-only surprise concert downtown yesterday and I just couldn't pass up the opportunity to attend. It is true that my excuse isn't traditionally considered a good one like that of the grieving student, but months ago I signaled to you quite clearly that an Exemplars concert would be the kind of thing that would lead me to turn a paper in late. You excused the grieving student because it is assumed in our culture that a death of a family member will give someone reason not to write a paper. But though we don't assume most people will not write a paper because of a concert you could have predicted an Exemplars concert would cause me not to write the paper. This excuse won't be used often by other students since to be successful one has to indicate a stance that would cause them to not write the paper under certain circumstances and then hope those circumstances arise."

Comment author: Yvain 21 April 2010 05:45:05PM 19 points [-]

If I were a teacher, I'd ideally allow this (and the sports fan too, for that matter, if I knew he was a big enough fan).

There's no problem in the ideal form, but I would anticipate a lot of problems in real life - other students not understanding exactly how much this person loved the Exemplars and misinterpreting it as "You get an exemption for going to a rock concert", other students trying to convince you they like [thing X] just as much as this guy liked the Exemplars and you having to judge lots of difficult cases to see if their fandom is truly as great as this guy's is and inevitably getting some of them wrong.

If I really knew what to do, this post would have been titled "Eight Short Studies On Excuses, Plus The Answers To Them"

Comment author: Yvain 20 April 2010 11:10:41PM 6 points [-]

I'm doing exactly what I would be doing if I had never found Less Wrong, but now I'm telling myself this is provably the best course because it will make me a lot of money which I can donate to the usual worthy projects. This argument raises enough red flags that I'm well aware of how silly it sounds, but I can't find any particular flaws in the logic.

Eight Short Studies On Excuses

210 Post author: Yvain 20 April 2010 11:01PM

The Clumsy Game-Player

You and a partner are playing an Iterated Prisoner's Dilemma. Both of you have publicly pre-committed to the tit-for-tat strategy. By iteration 5, you're going happily along, raking up the bonuses of cooperation, when your partner unexpectedly presses the "defect" button.

"Uh, sorry," says your partner. "My finger slipped."

"I still have to punish you just in case," you say. "I'm going to defect next turn, and we'll see how you like it."

"Well," said your partner, "knowing that, I guess I'll defect next turn too, and we'll both lose out. But hey, it was just a slipped finger. By not trusting me, you're costing us both the benefits of one turn of cooperation."

"True", you respond "but if I don't do it, you'll feel free to defect whenever you feel like it, using the 'finger slipped' excuse."

"How about this?" proposes your partner. "I promise to take extra care that my finger won't slip again. You promise that if my finger does slip again, you will punish me terribly, defecting for a bunch of turns. That way, we trust each other again, and we can still get the benefits of cooperation next turn."

You don't believe that your partner's finger really slipped, not for an instant. But the plan still seems like a good one. You accept the deal, and you continue cooperating until the experimenter ends the game.

After the game, you wonder what went wrong, and whether you could have played better. You decide that there was no better way to deal with your partner's "finger-slip" - after all, the plan you enacted gave you maximum possible utility under the circumstances. But you wish that you'd pre-committed, at the beginning, to saying "and I will punish finger slips equally to deliberate defections, so make sure you're careful."



The Lazy Student

You are a perfectly utilitarian school teacher, who attaches exactly the same weight to others' welfare as to your own. You have to have the reports of all fifty students in your class ready by the time midterm grades go out on January 1st. You don't want to have to work during Christmas vacation, so you set a deadline that all reports must be in by December 15th or you won't grade them and the students will fail the class. Oh, and your class is Economics 101, and as part of a class project all your students have to behave as selfish utility-maximizing agents for the year.

It costs your students 0 utility to turn in the report on time, but they gain +1 utility by turning it in late (they enjoy procrastinating). It costs you 0 utility to grade a report turned in before December 15th, but -30 utility to grade one after December 15th. And students get 0 utility from having their reports graded on time, but get -100 utility from having a report marked incomplete and failing the class.

If you say "There's no penalty for turning in your report after deadline," then the students will procrastinate and turn in their reports late, for a total of +50 utility (1 per student times fifty students). You will have to grade all fifty reports during Christmas break, for a total of - 1500 utility (-30 per report times fifty reports). Total utility is -1450.

So instead you say "If you don't turn in your report on time, I won't grade it." All students calculate the cost of being late, which is +1 utility from procrastinating and -100 from failing the class, and turn in their reports on time. You get all reports graded before Christmas, no students fail the class, and total utility loss is zero. Yay!

Or else - one student comes to you the day after deadline and says "Sorry, I was really tired yesterday, so I really didn't want to come all the way here to hand in my report. I expect you'll grade my report anyway, because I know you to be a perfect utilitarian, and you'd rather take the -30 utility hit to yourself than take the -100 utility hit to me."

You respond "Sorry, but if I let you get away with this, all the other students will turn in their reports late in the summer." She says "Tell you what - our school has procedures for changing a student's previously given grade. If I ever do this again, or if I ever tell anyone else about this, you can change my grade to a fail. Now you know that passing me this one time won't affect anything in the future. It certainly can't affect the past. So you have no reason not to do it." You believe her when she says she'll never tell, but you say "You made this argument because you believed me to be the sort of person who would accept it. In order to prevent other people from making the same argument, I have to be the sort of person who wouldn't accept it. To that end, I'm going to not accept your argument."

The Grieving Student

A second student comes to you and says "Sorry I didn't turn in my report yesterday. My mother died the other day, and I wanted to go to her funeral."

You say "Like all economics professors, I have no soul, and so am unable to sympathize with your loss. Unless you can make an argument that would apply to all rational actors in my position, I can't grant you an extension."

She says "If you did grant this extension, it wouldn't encourage other students to turn in their reports late. The other students would just say 'She got an extension because her mother died'. They know they won't get extensions unless they kill their own mothers, and even economics students aren't that evil. Further, if you don't grant the extension, it won't help you get more reports in on time. Any student would rather attend her mother's funeral than pass a course, so you won't be successfully motivating anyone else to turn in their reports early."

You think for a while, decide she's right, and grant her an extension on her report.

The Sports Fan

A third student comes to you and says "Sorry I didn't turn in my report yesterday. The Bears' big game was on, and as I've told you before, I'm a huge Bears fan. But don't worry! It's very rare that there's a game on this important, and not many students here are sports fans anyway. You'll probably never see a student with this exact excuse again. So in a way, it's not that different from the student here just before me, the one whose mother died."

You respond "It may be true that very few people will be able to say both that they're huge Bears fans, and that there's a big Bears game on the day before the report comes due. But by accepting your excuse, I establish a precedent of accepting excuses that are approximately this good. And there are many other excuses approximately as good as yours. Maybe someone's a big soap opera fan, and the season finale is on the night before the deadline. Maybe someone loves rock music, and there's a big rock concert on. Maybe someone's brother is in town that week. Practically anyone can come up with an excuse as good as yours, so if I accept your late report, I have to accept everyone's.

"The student who was here before you, that's different. We, as a society, already have an ordering in which a family member's funeral is one of the most important things around. By accepting her excuse, I'm establishing a precedent of accepting any excuse approximately that good, but almost no one will ever have an excuse that good. Maybe a few people who are really sick, someone struggling with a divorce or a breakup, that kind of thing. Not the hordes of people who will be coming to me if I give you your exemption."

The Murderous Husband

You are the husband of a wonderful and beautiful lady whom you love very much - and whom you just found in bed with another man. In a rage, you take your hardcover copy of Introduction To Game Theory and knock him over the head with it, killing him instantly (it's a pretty big book).

At the murder trial, you plead to the judge to let you go free. "Society needs to lock up murderers, as a general rule. After all, they are dangerous people who cannot be allowed to walk free. However, I only killed that man because he was having an affair with my wife. In my place, anyone would have done the same. So the crime has no bearing on how likely I am to murder someone else. I'm not a risk to anyone who isn't having an affair with my wife, and after this incident I plan to divorce and live the rest of my days a bachelor. Therefore, you have no need to deter me from future murders, and can safely let me go free."

The judge responds: "You make a convincing argument, and I believe that you will never kill anyone else in the future. However, other people will one day be in the position you were in, where they walk in on their wives having an affair. Society needs to have a credible pre-commitment to punishing them if they succumb to their rage, in order to deter them from murder."

"No," you say, "I understand your reasoning, but it won't work. If you've never walked in on your wife having an affair, you can't possibly understand the rage. No matter how bad the deterrent was, you'd still kill the guy."

"Hm," says the judge. "I'm afraid I just can't believe anyone could ever be quite that irrational. But I see where you're coming from. I'll give you a lighter sentence."

The Bellicose Dictator

You are the dictator of East Examplestan, a banana republic subsisting off its main import, high quality hypothetical scenarios. You've always had it in for your ancestral enemy, West Examplestan, but the UN has made it clear that any country in your region that aggressively invades a neighbor will be severely punished with sanctions and possible enforced "regime change." So you decide to leave the West alone for the time being.

One day, a few West Examplestanis unintentionally wander over your unmarked border while prospecting for new scenario mines. You immediately declare it a "hostile incursion" by "West Examplestani spies", declare war, and take the Western capital in a sneak attack.

The next day, Ban Ki-moon is on the phone, and he sounds angry. "I thought we at the UN had made it perfectly clear that countries can't just invade each other anymore!"

"But didn't you read our propaganda mouthpi...ahem, official newspaper? We didn't just invade. We were responding to Western aggression!"

"Balderdash!" says the Secretary-General. "Those were a couple of lost prospectors, and you know it!"

"Well," you say. "Let's consider your options. The UN needs to make a credible pre-commitment to punish aggressive countries, or everyone will invade their weaker neighbors. And you've got to follow through on your threats, or else the pre-commitment won't be credible anymore. But you don't actually like following through on your threats. Invading rogue states will kill a lot of people on both sides and be politically unpopular, and sanctions will hurt your economy and lead to heart-rending images of children starving. What you'd really like to do is let us off, but in a way that doesn't make other countries think they'll get off too.

"Luckily, we can make a credible story that we were following international law. Sure, it may have been stupid of us to mistake a few prospectors for an invasion, but there's no international law against being stupid. If you dismiss us as simply misled, you don't have to go through the trouble of punishing us, and other countries won't think they can get away with anything.

"Nor do you need to live in fear of us doing something like this again. We've already demonstrated that we won't go to war without a casus belli. If other countries can refrain from giving us one, they have nothing to fear."

Ban Ki-moon doesn't believe your story, but the countries that would bear the economic brunt of the sanctions and regime change decide they believe it just enough to stay uninvolved.

The Peyote-Popping Native

You are the governor of a state with a large Native American population. You have banned all mind-altering drugs, with the honorable exceptions of alcohol, tobacco, caffeine, and several others, because you are a red-blooded American who believes that they would drive teenagers to commit crimes.

A representative of the state Native population comes to you and says: "Our people have used peyote religiously for hundreds of years. During this time, we haven't become addicted or committed any crimes. Please grant us a religious exemption under the First Amendment to continue practicing our ancient rituals." You agree.

A leader of your state's atheist community breaks into your office via the ventilation systems (because seriously, how else is an atheist leader going to get access to a state governor?) and says: "As an atheist, I am offended that you grant exemptions to your anti-peyote law for religious reasons, but not for, say, recreational reasons. This is unfair discrimination in favor of religion. The same is true of laws that say Sikhs can wear turbans in school to show support for God, but my son can't wear a baseball cap in school to show support for the Yankees. Or laws that say Muslims can get time off state jobs to pray five times a day, but I can't get time off my state job for a cigarette break. Or laws that say state functions will include special kosher meals for Jews, but not special pasta meals for people who really like pasta."

You respond "Although my policies may seem to be saying religion is more important than other potential reasons for breaking a rule, one can make a non-religious case justifying them. One important feature of major world religions is that their rituals have been fixed for hundreds of years. Allowing people to break laws for religious reasons makes religious people very happy, but does not weaken the laws. After all, we all know the few areas in which the laws of the major US religions as they are currently practiced conflict with secular law, and none of them are big deals. So the general principle 'I will allow people to break laws if it is necessary to established and well-known religious rituals" is relatively low-risk and makes people happy without threatening the concept of law in general. But the general principle 'I will allow people to break laws for recreational reasons' is very high risk, because it's sufficient justification for almost anyone breaking any law."

"I would love to be able to serve everyone the exact meal they most wanted at state dinners. But if I took your request for pasta because you liked pasta, I would have to follow the general principle of giving everyone the meal they most like, which would be prohibitively expensive. By giving Jews kosher meals, I can satisfy a certain particularly strong preference without being forced to satisfy anyone else's."

The Well-Disguised Atheist

The next day, the atheist leader comes in again. This time, he is wearing a false mustache and sombrero. "I represent the Church of Driving 50 In A 30 Mile Per Hour Zone," he says. "For our members, going at least twenty miles per hour over the speed limit is considered a sacrament. Please grant us a religious exemption to traffic laws."

You decide to play along. "How long has your religion existed, and how many people do you have?" you ask.

"Not very long, and not very many people," he responds.

"I see," you say. "In that case, you're a cult, and not a religion at all. Sorry, we don't deal with cults."

"What, exactly, is the difference between a cult and a religion?"

"The difference is that cults have been formed recently enough, and are small enough, that we are suspicious of them existing for the purpose of taking advantage of the special place we give religion. Granting an exemption for your cult would challenge the credibility of our pre-commitment to punish people who break the law, because it would mean anyone who wants to break a law could just found a cult dedicated to it."

"How can my cult become a real religion that deserves legal benefits?"

"You'd have to become old enough and respectable enough that it becomes implausible that it was created for the purpose of taking advantage of the law."

"That sounds like a lot of work."

"Alternatively, you could try writing awful science fiction novels and hiring a ton of lawyers. I hear that also works these days."

Conclusion

In all these stories, the first party wants to credibly pre-commit to a rule, but also has incentives to forgive other people's deviations from the rule. The second party breaks the rules, but comes up with an excuse for why its infraction should be forgiven.

The first party's response is based not only on whether the person's excuse is believable, not even on whether the person's excuse is morally valid, but on whether the excuse can be accepted without straining the credibility of their previous pre-commitment.

The general principle is that by accepting an excuse, a rule-maker is also committing themselves to accepting all equally good excuses in the future. There are some exceptions - accepting an excuse in private but making sure no one else ever knows, accepting an excuse once with the express condition that you will never accept any other excuses - but to some degree these are devil's bargains, as anyone who can predict you will do this can take advantage of you.

These stories give an idea of excuses different from the one our society likes to think it uses, namely that it accepts only excuses that are true and that reflect well upon the character of the person giving the excuse. I'm not saying that the common idea of excuses doesn't have value - but I think the game theory view also has some truth to it. I also think the game theoretic view can be useful in cases where the common view fails. It can inform cases in law, international diplomacy, and politics where a tool somewhat stronger than the easily-muddled common view is helpful.

Comment author: Peter_Twieg 01 April 2010 05:33:15PM *  2 points [-]

I recently got into some arguments with foodies I know on the merits (or lack thereof) of organic / local / free-range / etc. food, and this is a topic where I find it very difficult to find sources of information that I trust as reflective of some sort of expert consensus (insofar as one can be said to exist.) Does anyone have any recommendations for books or articles on nutrition/health that holds up under critical scrutiny? I trust a lot of you as filters on these issues.

Comment author: Yvain 11 April 2010 01:47:55PM *  5 points [-]

There are lots of studies on the issue, and as usual most of them are bad and disagree with each other.

I tend to trust the one by the UK Food Standards Association because it's big and government-funded. Mayo Clinic agrees. I think there are a few studies that show organic foods do have lower pesticide levels than normal, but nothing showing that it actually leads to health benefits. Pesticides can cause some health problems in farmers, but they're receiving a bajillion times the dose of someone who just eats the occasional carrot. And some "organic pesticides" are just as bad as any synthetic ones. There's also a higher risk of getting bacterial infections from organic food.

Tastewise, a lot of organics people cite some studies showing that organic apples and other fruit taste better than conventional - I can't find the originals of these and there are equally questionable studies that say the opposite. Organic vegetables taste somewhere between the same and worse, even by organic peoples' admission. There's a pretty believable study showing conventional chicken tastes better than organic, and a more pop-sci study claiming the same thing about almost everything. I've seen some evidence that locally grown produce tastes better than imported, but that's a different issue than organic vs. non-organic and you have to make sure people aren't conflating them.

They do produce less environmental damage per unit land, but they produce much less food per unit land and so require more land to be devoted to agriculture. How exactly that works out in the end is complex economics that I can't navigate.

My current belief is that organics have a few more nutrients here and there but not enough to matter, are probably less healthy overall when you consider infection risk, and taste is anywhere from no difference to worse except maybe on a few limited fruits.

In response to Swimming in Reasons
Comment author: Yvain 11 April 2010 12:57:47PM 2 points [-]

Oh, good. That clarification makes things much better.

But there are still situations where you can believe a normative "for many reasons". The most important is probably that you can have many reasons to support something as well as to oppose it. For example, there may be five strong arguments to vote Demopublican, and only four equally strong arguments to vote Republicrat. In this case, if you ask me why I'm voting Demopublican, I'd have to say "for many reasons". Might sound like splitting hairs, but I think lots of real world cases fall into this category.

Another caveat is that this only applies to "randomly generated" alternatives. If I had to choose between a battle plan devised by a brilliant general, versus a battle plan devised by a moron, there would probably be "many reasons" to prefer the general's, and I would keep finding more the harder I looked. All of these would come back to one reason - "it was designed by a brilliant general", but the reason would be a fact about the plan that's inaccessible to anyone who doesn't know how the plan was designed, not a fact within the plan that could be discovered by looking it over. Again, may sound like splitting hairs, but a lot of real human issues could fall into this category, like using scientific versus alternative medicine.

Comment author: CannibalSmith 08 April 2010 12:34:32PM -1 points [-]

Help me, LessWrong. I want to build a case for

  1. Information is a terminal value without exception.
  2. All information is inherently good.
  3. We must gather and preserve information for its own sake.

These phrasings should mean the exact same thing. Correct me if they don't.

Elaboration: Most people readily agree that most information is good most of the time. I want to see if I can go all the way and build a convincing argument that all information is good all of the time, or as close to it as I can get. That misuse of information is problem about the misuser and not the information ("guns don't kill people"). Specific cases include: endangered species (DNA is best stored in living organisms), viruses (all three kinds), forbidden books, child pornography and other shocking information, free speech, Archive.org, The Rosetta Project, research on race.

Please post arguments and counterarguments in their own comments and separately from general discussion comments.

Comment author: Yvain 10 April 2010 12:04:04AM 4 points [-]

You probably don't mean trivial information eg the position of every oxygen atom in my room at this exact moment. But if you eliminate trivial information and concentrate only on useful information, you've turned it into a circular argument - all useful information is inherently useful.

Further, saying that we "must" gather and preserve information ignores opportunity costs. Sure, anything might eventually turn out to be useful, but at some point we have to say the resources invested in disk space would be better used somewhere else.

It sounds more like you're trying to argue that information can never be evil, but you can't even state that meaningfully without making a similar error. Certainly giving information to certain people can be evil (for example, giving Hitler the information on how to make a nuclear bomb).

See this discussion for why I think calling something like "information" good is a bad idea.

In response to Ureshiku Naritai
Comment author: Yvain 09 April 2010 07:19:10PM 4 points [-]

Thanks; I've bookmarked this and am going to try something like it. But I'm probably greedier than you are, so I've got to ask - do you think you could use the same techniques you used to raise your set point from 2 to 5, to raise your set point from 5 to 8?

Comment author: Eliezer_Yudkowsky 02 April 2010 05:43:01PM 3 points [-]

But that still doesn't need to be luck. I got my priors offa evolution and they are capable of noticing when something works or doesn't work a hundred times in a row. True, if I had a different prior, I wouldn't care about that either. But even so, that I have this prior is not a question of luck.

Comment author: Yvain 03 April 2010 10:30:13PM 8 points [-]

It is luck in a sense - every way that your opinion differs from someone else, you believe that factors outside of your control (your intelligence, your education, et cetera) have blessed you in such a way that your mind has done better than that poor person's.

It's just that it's not a problem. Lottery winners got richer than everyone else by luck, but that doesn't mean they're deluded in believing that they're rich. But someone who had only weak evidence ze won the lottery should be very skeptical. The real point of this quote is that being much less wrong than average is an improbable state, and you need correspondingly strong evidence to support the possibility. I think many of the people on this site probably do have some of that evidence (things like higher than average IQ scores would be decent signs of higher than normal probability of being right) but it's still something worth worrying about.

Comment author: Vladimir_Nesov 02 April 2010 08:22:25AM *  3 points [-]

Prior can't be judged. It's not assumed to be "correct". It's just the way you happen to process new info and make decisions, and there is no procedure to change the way it is from inside the system.

Comment author: Yvain 03 April 2010 10:24:24PM 0 points [-]

But one man's prior is another man's posterior: I can use the belief that a medical test is 90% specific when using it to determine whether a patient has a disease, but I arrived at my beliefs about that medical test through Bayesian processes - either logical reasoning about the science behind the test, or more likely trying the test on a bunch of people and using statistics to estimate a specificity.

So it may be mathematically wrong to tell me my 90% prior is false, but the 90% prior from the first question is the same 90% posterior from the second question, and it's totally kosher to say that the 90% posterior from the second question is wrong (and by extension, I'm using the "wrong prior")

The whole reflective consistency thing is that you shouldn't have "foundational priors" in the sense that they're not the posterior of anything. Every foundational prior gets checked by how well it accords with other things, and in that sense is sort of a posterior.

So I agree with cousin_it that it would be a problem if every Bayesian believed their prior to be correct (as in - they got the correct posterior yesterday to use as their prior today).

Comment author: Yvain 03 April 2010 10:01:29PM *  3 points [-]

I have recently been informed that there's a tradition here of closing certain stores this weekend to commemorate an agglomeration of the local resurrection myth with a previous pagan seasonal festival. I know some Waterstones in London are closed and others are open, but I can't find any information on that particular one. If it's closed, we can meet outside the front door and figure out where to go from there.

Comment author: Yvain 02 April 2010 12:46:16AM 23 points [-]

"Everyone thinks they've won the Magical Belief Lottery. Everyone thinks they more or less have a handle on things, that they, as opposed to the billions who disagree with them, have somehow lucked into the one true belief system."

-- R Scott Bakker, Neuropath

Comment author: Yvain 01 April 2010 08:11:37PM *  5 points [-]

The London meet is going ahead. Unless someone proposes a different time, or taw's old meetings are still going on and I just didn't know about them, it will be:

5th View cafe on top of Waterstone's bookstore near Piccadilly Circus Sunday, April 4 at 4PM

Roko, HumanFlesh, I've got your numbers and am hoping you'll attend and rally as many Londoners as you can.

EDIT: Sorry, Sunday, not Monday.

Comment author: rwallace 28 March 2010 03:21:56PM 1 point [-]

So we seem to have concluded that my Irish citizenship disproves the anthropic principle, and I can know this, but you cannot know it :-)

Comment author: Yvain 28 March 2010 04:08:48PM *  2 points [-]

As a matter of fact, I live in Ireland (although I'm a US citizen). That coincidence probably disproves some sort of important principle right there.

I think you've mentioned before that you live in Dublin; I live in Cork, so sadly we're a little too far to meet up for a chat one night.

Comment author: komponisto 27 March 2010 03:55:52PM 4 points [-]

Following bogus, I could imagine endorsing a weaker form of the argument: not that it's like nothing to be a bat, but that it's like less to be a bat than to be a human.

In fact, if you've ever wondered why you happen to be the person you are, and not someone else, it may be that the reflectivity you are displaying by asking this question puts you in a more-strongly-anthropically-weighted reference class.

Comment author: Yvain 28 March 2010 01:47:28PM 0 points [-]

Given 10 billion bats , that bats have been around for 50 million years, and bat generations taking let's say 5 years, and assuming that population has been stable for evolutionary history, we have a super rough estimate of something on the order of (10B * (50M/5)) = 100 quadrillion historical bats. I think a lot of anthropic calculations assume there have been 100 billion historical humans, so probability of being a human is 1/1 millionth the probability of being a bat.

I don't see a whole lot of difference between not having subjective experiences and having one one-millionth the subjective experience of a human. Once we expand this to all animals instead of just bats, the animals come out even worse.

Comment author: PhilGoetz 28 March 2010 12:29:08AM 7 points [-]

Can't I use the same reasoning to prove that non-Americans aren't conscious?

Comment author: Yvain 28 March 2010 01:37:41PM 0 points [-]

Not unless you have a strong reason to privilege the state of being an American as especially interesting. Otherwise, you're in the position Jordan mentioned of just knowing you're in one unexceptional condition out of many.

One thing you could say based on your being an American is that you have weak evidence that America is likely to be one of the more populous countries, and strong evidence that there's no country thousands or billions of times more populous than America. Both conclusions are correct.

And further, if a Luxembourgian posts a reply here saying "My Luxembourgian citizenship disproves the anthropic principle", that doesn't count, because you're not him and he's self-selected by posting here o_O

Comment author: PhilGoetz 27 March 2010 09:27:36PM *  11 points [-]

That's an interesting observation.

There's a problem in assuming that consciousness is a 0/1 property; that you're either conscious, or not.

There's another problem in assuming that YOU are a 0/1 property; that there is exactly one atomic "your consciousness".

Reflect on the discussion in the early chapters of Daniel Dennet's "Consciousness Explained", about how consciousness is not really a unitary thing, but the result of the interaction of many different processes.

An ant has fewer of these processes than you do. Instead of asking "What are the odds that 'I' ended up as me?", ask, "For one of these processes, what are the odds that it would end up in me, rather than in an ant?"

According to Wikipedia's entry on biomass, ants have 10-100 times the biomass of humans today.

According to Wikipedia's list of animals by neuron count, ants have 10,000 neurons.

According to that page, and this one, humans have 10^11 neurons.

Information is proportional not to the number of neurons, but to the number of patterns that can be stored in those neurons, which is likely somewhere between N and N^2. I'm gonna call it NlogN.

I weigh as much as 167,000 ants. Each of them has ~ 10,000 log(10,000) bits of info. I have ~ 10^11 log(10^11) bits of info. I contain as much information as 165 times my body-mass worth of ants.

So if we ignore how much longer ants have lived than humans, the odds are better that a random unit of consciousness today would turn up in a human, than in an ant.

(Also note that we can only take into account ants in the past, if reincarnation is false. If reincarnation is true, then you can't ask about the chances of you appearing in a different time. :) )

If you're gonna then say, "But let's not just compare ourselves to ants; let's ask about turning up in a human vs. turning up in any other species", then you have the dice-labelling problem argued below: You're claiming humans are the 1 on the die.

Comment author: Yvain 28 March 2010 01:32:44PM 0 points [-]

I weigh as much as 167,000 ants. Each of them has ~ 10,000 log(10,000) bits of info. I have ~ 10^11 log(10^11) bits of info. I contain as much information as 165 ants.

I'm not following your math here, and I'm especially not following the part where if a person contains as much information as 165 ants and there are 1 quadrillion ants and ~ 10 billion people, a given unit of information is more likely to end up in a human than in an ant. And since we do believe reincarnation is false, it's much worse than that, since ants have been around longer than humans.

Also, I have a philosophical objection with basing it on units consciousness. If we're to weight the chances of being a certain animal with the number of bits information they have, doesn't that imply we're working from a theory where "I" am a single bit of information? I'd much sooner say that I am all the information in my head equally, or an algorithm that processes that information, or at least not just a single bit of it.

In response to comment by [deleted] on It's not like anything to be a bat
Comment author: Jack 28 March 2010 09:35:55AM 2 points [-]

The question "Why am I not somebody else?" is nonsensical because for almost anybody I am somebody else.

More precisely: "I" refers to some numerically unique entity x. Thus "I is someone else" means x = -x which is an outright contradiction and we shouldn't waste our time asking why contradictions aren't the case.

Comment author: Yvain 28 March 2010 01:21:21PM 13 points [-]

It only sounds nonsensical because of the words in which it's asked. The question raised by anthropic reasoning isn't "why do I live in a time I call the present" (to which, as you say, the answer is linguistic - of course we'd call our time the present) but rather "why do I live in the year 2010?" or, most precisely of all, "Given that I have special access to the subjective experience of one being, why would that be the experience of a being born in the late 20th century, as opposed to some other time?"

That may still sound tautological - after all, if it wasn't the 20th century, it'd be somewhen else and we'd be asking the same question - but in fact it isn't. Consider these two questions:

  • Why am I made out of carbon, as opposed to helium?
  • Why do I live in the 20th century, as opposed to the 30th?

The correct answer to the second is not saying, "Well, if you were made out of helium, you could just ask why you were made out of helium, so it's a dumb question", it's pointing out the special chemical properties of carbon. Anthropic reasoning suggests that we can try doing the same to point out certain special properties of the 20th century.

The big difference is that the first question can be easily rephrased to "why are people made out of carbon and not of helium", but the second can't. But that difference isn't enough to make the second tautological or meaningless.

Comment author: Jonii 27 March 2010 03:42:09PM 12 points [-]

If you'd be any other animal on Earth, you wouldn't be considering what it would be like to be something else. Doomsday argument and arguments like it are usually formulated in a way "Of all the persons that could reason like me, only this small percentage ever were wrong". When animals are prevented, due to their neurological limitations, from reasoning as necessiated by the argument, they're not part of this consideration.

This doesn't mean that they're not sentient, it just means that by thinking about anthropic problems you're part of much narrower set of beings than just sentient ones.

Comment author: Yvain 27 March 2010 05:23:28PM 8 points [-]

Why not limit the set of people who could reason like me to "people who are using anthropic reasoning" and just assume people will stop using anthropic reasoning in the next hundred years? Is this a reductio ad absurdum, or do you think it's a valid conclusion?

It's not like anything to be a bat

15 Post author: Yvain 27 March 2010 02:32PM

...at least not if you accept a certain line of anthropic argument.

Thomas Nagel famously challenged the philosophical world to come to terms with qualia in his essay "What is it Like to Be a Bat?". Bats, with sensory systems so completely different from those of humans, must have exotic bat qualia that we could never imagine. Even if we deduce all the physical principles behind echolocation, even if we could specify the movement of every atom in a bat's senses and nervous system that represents its knowledge of where an echolocated insect is, we still have no idea what it's like to feel a subjective echolocation quale.

Anthropic reasoning is the idea that you can reason conditioning on your own existence. For example, the Doomsday Argument says that you would be more likely to exist in the present day if the overall number of future humans was medium-sized instead of humongous, therefore since you exist in the present day, there must be only a medium-sized number of future humans, and the apocalypse must be nigh, for values of nigh equal to "within a few hundred years or so".

The Buddhists have a parable to motivate young seekers after enlightenment. They say - there are zillions upon zillions of insects, trillions upon trillions of lesser animals, and only a relative handful of human beings. For a reincarnating soul to be born as a human being, then, is a rare and precious gift, and an opportunity that should be seized with great enthusiasm, as it will be endless eons before it comes around again.

Whatever one thinks of reincarnation, the parable raises an interesting point. Considering the vast number of non-human animals compared to humans, the probability of being a human is vanishingly low. Therefore, chances are that if I could be an animal, I would be. This makes a strong anthropic argument that it is impossible for me to be an animal.

The phrase "for me to be an animal" may sound nonsensical, but "why am I me, rather than an animal?" is not obviously sillier than "why am I me, rather than a person from the far future?". If the doomsday argument is sufficient to prove that some catastrophe is preventing me from being one of a trillion spacefaring citizens of the colonized galaxy, this argument hints that something is preventing me from being one of a trillion bats or birds or insects.

And this could be that animals lack subjective experience. This would explain quite nicely why I'm not an animal: because you can't be an animal, any more than you can be a toaster. So Thomas Nagel can stop worrying about what it's like to be a bat, and the rest of us can eat veal and foie gras guilt-free.

But before we break out the dolphin sausages - this is a pretty weird conclusion. It suggests there's a qualitative and discontinuous difference between the nervous system of other beings and our own, not just in what capacities they have but in the way they cause experience. It should make dualists a little bit happier and materialists a little bit more confused (though it's far from knockout proof of either).

The most significant objection I can think of is that it is significant not that we are beings with experiences, but that we know we are beings with experiences and can self-identify as conscious - a distinction that applies only to humans and maybe to some species like apes and dolphins who are rare enough not to throw off the numbers. But why can't we use the reference class of conscious beings if we want to? One might as well consider it significant only that we are beings who make anthropic arguments, and imagine there will be no Doomsday but that anthropic reasoning will fall out of favor in a few decades.

But I still don't fully accept this argument, and I'd be pretty happy if someone could find a more substantial flaw in it.

More thoughts on assertions

22 Post author: Yvain 25 March 2010 01:39AM

Response to: The "show, don't tell" nature of argument

Morendil says not to trust simple assertions. He's right, for the certain class of simple assertions he's talking about. But in order to see why, let's look at different types of assertions  and see how useful it is to believe them.

Summary:
- Hearing an assertion can be strong evidence if you know nothing else about the proposition in question.
- Hearing an assertion is not useful evidence if you already have a reasonable estimate of how many people do or don't believe the proposition.
- An assertion by a leading authority is stronger than an assertion by someone else.
- An assertion plus an assertion that there is evidence makes no factual difference, but is a valuable signal.

Unsupported assertions about non-controversial topics

Consider my assertion: "The Wikipedia featured article today is on Uriel Sebree". Even if you haven't checked Wikipedia today and have no evidence on this topic, you're likely to believe me. Why would I be lying?

This can be nicely modeled in Bayesian terms - you start with a prior evenly distributed across Wikipedia topics, the probability of me saying this conditional on it being false is pretty low, and the probability of me saying it conditional on it being true is pretty high. So noting that I said it nicely concentrates probability mass in the worlds where it's true. You're totally justified in believing it. The key here is that you have no reason to believe there's a large group of people who go around talking about Uriel Sebree being on Wikipedia regardless of whether or not he really is.

Unsupported assertions about controversial topics

The example given in Morendil's post is that some races are biologically less intelligent than others. Let's say you have no knowledge of this whatsoever. You're so naive you don't even realize it might be controversial. In this case, someone who asserts "some races are biologically less intelligent than others" is no less believable than someone who asserts "some races have slightly different frequencies of pancreatic cancer than others." You'd accept the second as the sort of boring but reliable biological fact that no one is particularly prone to lie about, and you'd do the same with the first.

Now let's say you're familiar with controversies in sociology and genetics, you already know that some people believe some races are biologically more intelligent, and other people don't. Let's say you gauge the people around you and find that about 25% of people agree with the statement and 75% disagree.

This survey could be useful. You have to ask yourself - is this statement about race and genetics more likely to have the support of a majority of people in a world where it's true than in a world where it's false? "No" is a perfectly valid answer here - you might think people are so interested in signalling that they're not racist that they'll completely suspend their rational faculties. But "yes" is also a valid answer here if you think that the people around you have reasonably intelligent opinions on the issue. This would be a good time to increase your probability that it's true.

Now I, a perfectly average member of the human race, make the assertion that I believe that statement. But from your survey, you already have information that negates any evidence from my belief - that given that the statement is false and there's a 25% belief rate, there's a 25% chance I would agree with it, and given that the statement is true and there's a 25% belief rate, there's a 25% chance I would agree with it. If you've already updated on your survey, my assertion is equally likely in both conditions and doesn't shift probability one way or the other.

Unsupported assertions on extremely unusual topics

There is a case, I think, in which a single person asserting ze believes something can increase your probability. Imagine that I say, truthfully, that I believe that a race of otter-people from Neptune secretly controls the World Cup soccer tournament. If you've never heard this particular insane theory before, your estimate of the number of people who believed it was probably either zero, or so low that you wouldn't expect anyone you actually meet (even for values of "meet" including online forums) to endorse it. My endorsing it actually raises your estimate of the percent of the human race who endorse it, and this should raise your probability of it being true. Clearly, it should not raise it very much, and it need not necessarily raise it at all to the degree that you can prove that I have reasons other than truth for making the assertion (in this case, most of the probability mass generated by the assertion would leak off into the proposition that I was insane) but it can raise it a little bit.

Unsupported assertions by important authorities

This effect becomes more important when the person involved has impressive credentials. If someone with a Ph.D in biology says that race plays a part in intelligence, this could shift your estimate. In particular, it would shift it if you previously thought the race-intelligence connection was such a fringe theory that they would be unlikely to get even one good biologist on their side. But if you already knew that this theory was somewhat mainstream and had at least a tiny bit of support from the scientific community, it would be giving no extra information. Consider this the Robin Hanson Effect, because a lot of the good Robin Hanson does comes from being a well-credentialed guy with a Ph.D willing to endorse theories that formerly sounded so crazy that people would not have expected even one Ph.D to endorse them.

In cases of the Hanson Effect, the way you found out about the credentialled supporter is actually pretty important. If you Googled "Ph.D who supports transhumanism" and found Robin's name, then all it tells you is that there is at least one Ph.D who supports transhumanism. But if you were at a bar, and you found out the person next to you was a Ph.D, and you asked zir out of the blue if ze supported transhumanism, and ze said yes, then you know that there are enough Ph.Ds who support transhumanism that randomly running into one at the bar is not that uncommon an event.

An extreme case of the Hanson Effect is hearing that the world's top expert supports something. If there's only one World's Top Expert, then that person's opinion is always meaningful. This is why it was such a big deal when Watson came out in favor of a connection between race and intelligence. Now, I don't know if Watson actually knows anything about human genetic variation. He could have just had one clever insight about biochemistry way back when, and be completely clueless around the rest of the field. But if we imagine he really is the way his celebrity status makes him seem - the World's Top Expert in the field of genetics - then his opinion carries special weight for two reasons: first of all, it's the only data point we have in the field of "what the World's Top Expert thinks", and second, it suggests that a large percentage of the rest of the scientific community agrees with him (his status as World's Top Expert makes him something of a randomly chosen data point, and it would be very odd if we randomly pick the only data point that shares this opinion).

Assertions supported by unsupported claims of "evidence"

So much for completely unsupported assertions. Seeing as most people are pretty good at making up "evidence" that backs their pet beliefs, does it add anything to say "...and I arrived at this conclusion using evidence" if you refuse to say what the evidence is?

Well, it's a good signal for sanity. Instead of telling you only that at least one person believes in this hypothesis, you now know that at least one person who is smart enough to understand that ideas require evidence believes it.

This is less useful than it sounds. Disappointingly, there are not too many ideas that are believed solely by stupid people. As mentioned before, even creationism can muster a list of Ph.Ds who support it. When I was much younger, I was once quite impressed to hear that there were creationist Ph.Ds with a long list of scientific accomplishments in various fields. Since then, I learned about compartmentalization. So all that this "...and I have evidence for this proposition" can do on a factual level is highlight the existence of compartmentalization for people who weren't already aware of it.

But on a nonfactual level...again, it signals sanity. The difference betwee "I believe some races are less intelligent than others" and "I believe some races are less intelligent than others, and I arrived at this conclusion using evidence" is that the second person is trying to convince you ze's not some random racist with an axe to grind, ze's an amateur geneticist addressing an interesting biological question. I don't evaluate the credibility of the two statements any differently, but I'd much rather hang out with the person who made the second one (assuming ze wasn't lying or trying to hide real racism behind a scientific veneer).

Keep in mind that most communication is done not to convince anyone of anything, but to signal the character of the person arguing (source: I arrived at this conclusion using evidence). One character signal may interfere with other character signals, and "I arrived at this belief through evidence" can be a powerful backup. I have a friend who's a physics Ph.D, an evangelical Christian with an strong interest in theology, and an American living abroad. If he tries to signal that he's an evangelical Christian, he's very likely to get shoved into the "redneck American with ten guns and a Huckabee bumper sticker" box unless he immediately adds something like "and I base this belief on sound reasoning." That is one very useful signal there, and if he hadn't given it, I probably would have never bothered talking to him further. It's not a signal that his beliefs are actually based on sound reasoning, but it's a signal that he's the kind of guy who realizes beliefs should be based on that sort of thing and is probably pretty smart.

You can also take this the opposite way. There's a great Dilbert cartoon where Dilbert's date says something like "I know there's no scientific evidence that crystals can heal people, but it's my point of view that they do." This is a different signal; something along the lines of "I'd like to signal my support for New Agey crystal medicine, but don't dock me points for ignoring the scientific evidence against it." This is more of a status-preserving manuever than the status-claiming "I have evidence for this" one, but astoundingly it seems to work pretty well (except on Dilbert, who responded, "When did ignorance become a point of view?")

Comment author: Yvain 27 March 2010 12:29:50PM 6 points [-]

I wonder if there'd be a difference between the survey as written (asking what a pen would do on the moon, and then offering a chance to change the answer based on Apollo astronauts) vs. a survey in which someone asked "Given that the Apollo astronauts walked on the moon, what do you think would have happened if they'd dropped a pen?"

The first method makes someone commit to a false theory, and then gives them information that challenges the theory. People could passively try to fit the astronaut datum into their current working theory, or they could actively view it as an outside attack on their position which they had to defend against. Maybe if the students had given people the information about the astronauts first, the respondents would have applied the cross-domain knowledge more successfully.

But I totally sympathize with you about the occasional virtues of compartmentalization. The worst field I've ever found for this is health and medicine. You learn that some vitamin is an antioxidant, then you learn that some disease is caused by oxidation, you make the natural assumption that the vitamin would help cure the disease, and then a study comes out saying there's no relationship at all.

Comment author: Johnicholas 26 March 2010 10:59:49AM *  4 points [-]

What about casual use of poorly chosen examples reinforcing cultural concepts such as sexism? I'm referencing this paper. Summary: Example sentences in linguistics far more often have males verbing females than females verbing males.

There are a lot of questions which (to the best of my understanding) are still up in the air. Yvain's casual use of the controversial race/intelligence connection as an example at best glosses over these questions, and at worst subtly signals presumed answers to the questions without offering actual evidence. (Just like the males verbing females examples subtly signals some sort of cultural sexism.)

Questions like: Is intelligence a stable, innate quality? Is intelligence the same thing as IQ? Is intelligence a sharp, rigid concept, suitable for building theory-structures with? Is intelligence strongly correlated to IQ? Is race a sharp, rigid concept, suitable for building theory-structures with? Is IQ strongly correlated to self-identified race? Is race strongly correlated to genetics? Is the best explanation of these correlations that genetics strongly influences intelligence? Is the state of the scientific evidence settled enough that people ought to be taking the research and applying it to daily lives or policy decisions?

My take on it is that intelligence is a dangerously fuzzy concept, sliding from a general "tendency to win" on the one hand, to a simple multiple-choice questionaire on the other, all the while scattering assumptions of that it's innate, culture-free and unchangeable through your mind. Race is a dangerous concept too, with things like the one-drop rule confusing the connection to genetics and the fact that (according to the IAT) essentially everyone is a little bit racist, which has to affect your thinking about race. Thirdly, there's a very strong tendency for the racist/anti-racist politicals to hijack tentative scientific results and use them as weapons, which clouds the waters and makes everything a bit more explosive.

The post has an overt message (regarding assertions) and a covert signalling message, something about the putative race/intelligence connection. My sense of the lesswrong aesthetic has been wrong before, but I think we would prefer one-level to two-level posts (explicitness as a rationalist virtue).

Comment author: Yvain 26 March 2010 08:34:10PM *  4 points [-]

I used the race-intelligence connection as an example because it was the example used in Morendil's post to which this was a reply, which itself used it because it was the topic of the comment Morendil noticed was doing it wrong. I probably should have made this clearer; if you weren't following the history, it does look like a really really badly chosen example.

I personally have no strong opinion one way or the other. IQ is very hard to pin down genetically, seems to be distributed across hundreds of genes, and would probably be much harder to change than things like skin pigmentation or lactose tolerance. But generally I classify it in the category of "things whose political flammability is so far out of proportion to its actual importance that it's not worth thinking about unless you're itching for a fight".

In response to The Shabbos goy
Comment author: Yvain 26 March 2010 08:07:40PM *  30 points [-]

Presumably they also went to non-Jewish doctors if they had a medical emergency on the Sabbath.

Nitpick: Jewish law permits and in fact requires the breaking of Biblical commandments if someone's life is in danger (exceptions: idolatry, murder and breaking sexual taboos). In fact, there's an old Jewish joke that I think gets at some of the moral weirdness you're bringing up here:

A Jew and his Christian friend are sitting down for dinner at a restaurant. The Christian recommends the cheeseburger, but the Jew says that, since he keeps kosher, he's forbidden from eating any meal with both meat and dairy.

The Christian asks whether the Jew would eat a cheeseburger if his life depended on it, and the Jew says that Jewish law allows someone to break a ritual law if zir life is in danger. So the Christian draws a handgun, points it at his Jewish friend, and says "Order the cheeseburger, or I'll shoot!"

After dinner, the Christian puts his gun away and apologizes to the Jew. "You know I never would have shot you," he says. "I just wanted to see what you would do. Please forgive me."

"Forgive you?!" asks the Jew. "Of course I won't forgive you! Why didn't you make me order the bacon burger?"

In conclusion, if your morals make you wish someone would come along at gunpoint and force you to break your moral system, you may also be in error :)

Comment author: Morendil 25 March 2010 02:10:36AM 1 point [-]

Keep in mind that most communication is done not to convince anyone of anything, but to signal the character of the person arguing (source: I arrived at this conclusion using evidence)

I'm tempted to downvote the post solely for this. I'll hold off until I've had a chance to write more about (first) status, then (possibly) signalling. "Signalling" is the other concept that I'm seeing used a lot around here in a way that manages to consistently confuse me.

Meanwhile, let me just be on record as being very, very skeptical of the above assertion.

Comment author: Yvain 26 March 2010 12:15:18AM 2 points [-]

I was hoping that would be identifiable as a joke. You know, article about making assertions and then saying you have evidence without giving any, then making a controversial assertion and saying I have evidence without giving any? Sort of ironic, ha ha? No? Okay, sorry, I won't do it again.

Comment author: Nanani 25 March 2010 12:32:24AM 3 points [-]

Might it be as simple as being Special? Someone with an unusual medical problem is deserving of extra attention by virtue of having it.

Comment author: Yvain 26 March 2010 12:12:20AM 1 point [-]

That goes well with common sense, but how would you go about giving it predictive value? I mean, if I were to announce to a bunch of people one day that I had urinary incontinence, I imagine that would lower my status quite a bit. Certainly from a evo psych theoretical standpoint, it's odd to gain status by talking about how unhealthy and unfit you are.

I like what Andy Wood said about it being contextual, and hopefully if Kaj writes a summary of that chapter that'll help explain how the context works.

More thoughts on assertions

22 Post author: Yvain 25 March 2010 01:39AM

Response to: The "show, don't tell" nature of argument

Morendil says not to trust simple assertions. He's right, for the certain class of simple assertions he's talking about. But in order to see why, let's look at different types of assertions  and see how useful it is to believe them.

Summary:
- Hearing an assertion can be strong evidence if you know nothing else about the proposition in question.
- Hearing an assertion is not useful evidence if you already have a reasonable estimate of how many people do or don't believe the proposition.
- An assertion by a leading authority is stronger than an assertion by someone else.
- An assertion plus an assertion that there is evidence makes no factual difference, but is a valuable signal.

Unsupported assertions about non-controversial topics

Consider my assertion: "The Wikipedia featured article today is on Uriel Sebree". Even if you haven't checked Wikipedia today and have no evidence on this topic, you're likely to believe me. Why would I be lying?

This can be nicely modeled in Bayesian terms - you start with a prior evenly distributed across Wikipedia topics, the probability of me saying this conditional on it being false is pretty low, and the probability of me saying it conditional on it being true is pretty high. So noting that I said it nicely concentrates probability mass in the worlds where it's true. You're totally justified in believing it. The key here is that you have no reason to believe there's a large group of people who go around talking about Uriel Sebree being on Wikipedia regardless of whether or not he really is.

Unsupported assertions about controversial topics

The example given in Morendil's post is that some races are biologically less intelligent than others. Let's say you have no knowledge of this whatsoever. You're so naive you don't even realize it might be controversial. In this case, someone who asserts "some races are biologically less intelligent than others" is no less believable than someone who asserts "some races have slightly different frequencies of pancreatic cancer than others." You'd accept the second as the sort of boring but reliable biological fact that no one is particularly prone to lie about, and you'd do the same with the first.

Now let's say you're familiar with controversies in sociology and genetics, you already know that some people believe some races are biologically more intelligent, and other people don't. Let's say you gauge the people around you and find that about 25% of people agree with the statement and 75% disagree.

This survey could be useful. You have to ask yourself - is this statement about race and genetics more likely to have the support of a majority of people in a world where it's true than in a world where it's false? "No" is a perfectly valid answer here - you might think people are so interested in signalling that they're not racist that they'll completely suspend their rational faculties. But "yes" is also a valid answer here if you think that the people around you have reasonably intelligent opinions on the issue. This would be a good time to increase your probability that it's true.

Now I, a perfectly average member of the human race, make the assertion that I believe that statement. But from your survey, you already have information that negates any evidence from my belief - that given that the statement is false and there's a 25% belief rate, there's a 25% chance I would agree with it, and given that the statement is true and there's a 25% belief rate, there's a 25% chance I would agree with it. If you've already updated on your survey, my assertion is equally likely in both conditions and doesn't shift probability one way or the other.

Unsupported assertions on extremely unusual topics

There is a case, I think, in which a single person asserting ze believes something can increase your probability. Imagine that I say, truthfully, that I believe that a race of otter-people from Neptune secretly controls the World Cup soccer tournament. If you've never heard this particular insane theory before, your estimate of the number of people who believed it was probably either zero, or so low that you wouldn't expect anyone you actually meet (even for values of "meet" including online forums) to endorse it. My endorsing it actually raises your estimate of the percent of the human race who endorse it, and this should raise your probability of it being true. Clearly, it should not raise it very much, and it need not necessarily raise it at all to the degree that you can prove that I have reasons other than truth for making the assertion (in this case, most of the probability mass generated by the assertion would leak off into the proposition that I was insane) but it can raise it a little bit.

Unsupported assertions by important authorities

This effect becomes more important when the person involved has impressive credentials. If someone with a Ph.D in biology says that race plays a part in intelligence, this could shift your estimate. In particular, it would shift it if you previously thought the race-intelligence connection was such a fringe theory that they would be unlikely to get even one good biologist on their side. But if you already knew that this theory was somewhat mainstream and had at least a tiny bit of support from the scientific community, it would be giving no extra information. Consider this the Robin Hanson Effect, because a lot of the good Robin Hanson does comes from being a well-credentialed guy with a Ph.D willing to endorse theories that formerly sounded so crazy that people would not have expected even one Ph.D to endorse them.

In cases of the Hanson Effect, the way you found out about the credentialled supporter is actually pretty important. If you Googled "Ph.D who supports transhumanism" and found Robin's name, then all it tells you is that there is at least one Ph.D who supports transhumanism. But if you were at a bar, and you found out the person next to you was a Ph.D, and you asked zir out of the blue if ze supported transhumanism, and ze said yes, then you know that there are enough Ph.Ds who support transhumanism that randomly running into one at the bar is not that uncommon an event.

An extreme case of the Hanson Effect is hearing that the world's top expert supports something. If there's only one World's Top Expert, then that person's opinion is always meaningful. This is why it was such a big deal when Watson came out in favor of a connection between race and intelligence. Now, I don't know if Watson actually knows anything about human genetic variation. He could have just had one clever insight about biochemistry way back when, and be completely clueless around the rest of the field. But if we imagine he really is the way his celebrity status makes him seem - the World's Top Expert in the field of genetics - then his opinion carries special weight for two reasons: first of all, it's the only data point we have in the field of "what the World's Top Expert thinks", and second, it suggests that a large percentage of the rest of the scientific community agrees with him (his status as World's Top Expert makes him something of a randomly chosen data point, and it would be very odd if we randomly pick the only data point that shares this opinion).

Assertions supported by unsupported claims of "evidence"

So much for completely unsupported assertions. Seeing as most people are pretty good at making up "evidence" that backs their pet beliefs, does it add anything to say "...and I arrived at this conclusion using evidence" if you refuse to say what the evidence is?

Well, it's a good signal for sanity. Instead of telling you only that at least one person believes in this hypothesis, you now know that at least one person who is smart enough to understand that ideas require evidence believes it.

This is less useful than it sounds. Disappointingly, there are not too many ideas that are believed solely by stupid people. As mentioned before, even creationism can muster a list of Ph.Ds who support it. When I was much younger, I was once quite impressed to hear that there were creationist Ph.Ds with a long list of scientific accomplishments in various fields. Since then, I learned about compartmentalization. So all that this "...and I have evidence for this proposition" can do on a factual level is highlight the existence of compartmentalization for people who weren't already aware of it.

But on a nonfactual level...again, it signals sanity. The difference betwee "I believe some races are less intelligent than others" and "I believe some races are less intelligent than others, and I arrived at this conclusion using evidence" is that the second person is trying to convince you ze's not some random racist with an axe to grind, ze's an amateur geneticist addressing an interesting biological question. I don't evaluate the credibility of the two statements any differently, but I'd much rather hang out with the person who made the second one (assuming ze wasn't lying or trying to hide real racism behind a scientific veneer).

Keep in mind that most communication is done not to convince anyone of anything, but to signal the character of the person arguing (source: I arrived at this conclusion using evidence). One character signal may interfere with other character signals, and "I arrived at this belief through evidence" can be a powerful backup. I have a friend who's a physics Ph.D, an evangelical Christian with an strong interest in theology, and an American living abroad. If he tries to signal that he's an evangelical Christian, he's very likely to get shoved into the "redneck American with ten guns and a Huckabee bumper sticker" box unless he immediately adds something like "and I base this belief on sound reasoning." That is one very useful signal there, and if he hadn't given it, I probably would have never bothered talking to him further. It's not a signal that his beliefs are actually based on sound reasoning, but it's a signal that he's the kind of guy who realizes beliefs should be based on that sort of thing and is probably pretty smart.

You can also take this the opposite way. There's a great Dilbert cartoon where Dilbert's date says something like "I know there's no scientific evidence that crystals can heal people, but it's my point of view that they do." This is a different signal; something along the lines of "I'd like to signal my support for New Agey crystal medicine, but don't dock me points for ignoring the scientific evidence against it." This is more of a status-preserving manuever than the status-claiming "I have evidence for this" one, but astoundingly it seems to work pretty well (except on Dilbert, who responded, "When did ignorance become a point of view?")

Comment author: Yvain 24 March 2010 10:07:33PM 1 point [-]

Interesting to say that having an unusual medical problem raises someone's status. It sounds intuitively right, but I don't think any theory of status discussed here so far quite covers it.

Comment author: Yvain 24 March 2010 01:44:45AM *  2 points [-]

I'm doing my best here to approach what "a tiny piece of your soul" might mean. But no matter; perhaps you have a better idea of what that is. In any case, suppose you somehow isolated this tiny fraction of mind or spirit, and took it out of the context of all the countless other details we didn't look at. Now it's disconnected from all that other stuff: vision, balance, nachos, nuances of empathy…Given modern science, there is something more you can say about a particle besides the geometry and equations that describe it, something which connects it to the direct, cogito-ergo-sum style knowledge we all enjoy: whatever it is, a particle is a one thousand-trillion-trillionth of a you. Yes, you, your essence. Yes, science can now describe, in more or less complete detail, a one thousand-trillion-trillionth of a "soul".

Um...originally I thought this argument was skipping over the whole "how do particles lead to conscious experience" issue, but looking over it again I think I detect a hint of panpsychism - that all particles are conscious and the fact that you're conscious is an effect of the particles that make you up. If that was your intent, it definitely deserves a more complete treatment than a few sentences buried in an article about convincing nonmaterialists. If that wasn't your intent, how does this explain the apparent gap between physical matter and conscious experience at all?

Comment author: DanielVarga 15 March 2010 04:51:30PM 6 points [-]

"Stop! We have reached the limits of what rectal probing can teach us." One of my favourite Simpsons quotes.

Comment author: Yvain 17 March 2010 06:24:14PM 2 points [-]

Also, this

Comment author: Yvain 16 March 2010 09:28:00PM 43 points [-]

Two more non-trustworthy indicators:

  • Ask the person in question which of the several ridiculous ideas they reject they find least ridiculous - for example "Which do you think is more likely to be true - astrology, or UFOs?" I've found people trying to signal affiliation have a hard time with this sort of question and will even be flustered by it, saying something along the lines of "They're both stupid" or "Is this some sort of trick to make me sound like I believe a crazy idea?". A rationalist will say something more like "Well, I don't believe either, but UFOs at least make sense with our idea of the universe, whereas astrology is just plain crazytalk" (or ze may refuse to answer on the grounds that you're wasting zir time; it's not a perfect test).

  • Observe the circumstances in which the person involved brings up the belief. If they just go to atheist forums and say "Man, those religious people sure are stupid," higher probability of signaller. If they actively talk to religious people, try to use atheism as a starting point for building new ideas, and don't bring it up much when it's not relevant, higher probability they believe it for the right reasons.

Comment author: Roko 12 March 2010 06:52:26PM 0 points [-]

If I am around I'll come. CipherGoth will probably be interested too.

Comment author: Yvain 12 March 2010 07:47:28PM 0 points [-]

Why don't you PM me your phone number and/or email address and we can try to arrange something?

Comment author: Yvain 11 March 2010 10:49:31PM 8 points [-]

Repost from last open thread in the desperate hope that the lack of interest was only due to people not seeing it all the way at the bottom:

I'll be in London on April 4th and very interested in meeting any Less Wrongers who might be in the area that day. If there's a traditional LW London meetup venue, remind me what it is; if not, someone who knows the city suggest one and I'll be there. On an unrelated note, sorry I've been and will continue to be too busy/akratic to do anything more than reply to a couple of my PMs recently.

Comment author: Yvain 07 March 2010 10:53:31PM *  0 points [-]

I'll be in London on April 4th and very interested in meeting any Less Wrongers who might be in the area that day. If there's a traditional LW London meetup venue, remind me what it is; if not, someone who knows the city suggest one and I'll be there. On an unrelated note, sorry I've been and will continue to be too busy/akratic to do anything more than reply to a couple of my PMs recently.

Comment author: Yvain 01 February 2010 12:39:10PM 6 points [-]

Fun sneaky confidence exercise (reasons why exercise is fun and sneaky to be revealed later):

Please reply to this comment with your probability level that the "highest" human mental functions, such as reasoning and creative thought, operate solely on a substrate of neurons in the physical brain.

Comment author: Yvain 01 February 2010 12:24:51PM 16 points [-]

In our public medical personas, we often act as though morality consisted only in following society's conventions: we do this not so much out of laziness but because we recognize that it is better that the public think of doctors as old-fashioned or stupid, than that they should think us evil.

-- The Oxford Handbook of Clinical Medicine

Comment author: Yvain 01 February 2010 12:21:53PM *  48 points [-]

On utility:

culturejammer: you know what pennies are AWESOME for?

culturejammer: throwing at cats

culturejammer: it only costs a single penny

culturejammer: and they'll either chase it, or get hit by it and look pissed off

culturejammer: i now use that system to value prices of things

culturejammer: for example, a thirty dollar game has to be at least as awesome as three thousand catpennies

--bash.org

In response to Are wireheads happy?
Comment author: dclayh 01 January 2010 06:24:07PM *  8 points [-]

For the record, the actual Landsburg quote is

In one recent survey, 39 percent of New Yorkers said they would leave the city "if they could"! Every one of them was in New York on the day of the interview, so we know that at a minimum, 39 percent of New Yorkers lie to pollsters.

page 63 of his latest book The Big Questions.

Although I'm generally a big fan of Landsburg, this seems much more a case of confusion over what "leave the city" and "if you can" mean than one of lying.

Comment author: Yvain 01 January 2010 08:06:12PM 4 points [-]

I was reading More Sex is Safer Sex, so he must like using this anecdote a lot.

Comment author: Tyrrell_McAllister 01 January 2010 05:45:18PM 1 point [-]

You are equivocating on the term 'utility' here, as have so many other commenters before in this forum.

That seems to me to be an unfair reading. Nowhere does Yvain say that he's using the axiomatic theory of utility. It's true that he writes, "By definition, if you choose X over Y, then X is a higher utility option than Y." However, this definition can hold in other theoretical frameworks besides axiomatic utility theory. In particular, the definition plausibly holds in the framework used by some ethical utilitarians. Yvain can therefore be read as using the same definition for utility throughout.

Comment author: Yvain 01 January 2010 05:56:48PM 8 points [-]

I accept Benthamite's criticism as valid. It may not be obvious from the text, but in my mind I was definitely equivocating.

If we can't use preference to determine ethical utility, it makes ethical utilitarianism a lot harder, but that might be something we have to live with. I don't remember very much about Coherent Extrapolated Volition, but my vague memories say it makes that a lot harder too.

Comment author: Kaj_Sotala 01 January 2010 05:13:19PM 6 points [-]

Suppose you could find out the exact outcome (up to the point of reading the alternate history equivalent of Wikipedia, history books etc.) of changing the outcome of a single historical event. What would that event be?

Note that major developments like "the Roman empire would never have fallen" or "the Chinese wouldn't have turned inwards" involve multiple events, not just one.

Comment author: Yvain 01 January 2010 05:53:48PM *  15 points [-]

So many. I can't limit it to one, but my top four would be "What if Mohammed had never been born?", "What if Julian the Apostate had succeeded in stamping out Christianity?" and "What if Thera had never blown and the Minoans had survived?" and "What if Alexander the Great had lived to a ripe old age?"

The civilizations of the Near East were fascinating, and although the early Islamic Empire was interesting in its own right it did a lot to homogenize some really cool places. It also dealt a fatal wound to Byzantium as well. If Mohammed had never existed, I would look forward to reading about the Zoroastrian Persians, the Byzantines, and the Romanized Syrians and Egyptians surviving much longer than they did.

The Minoans were the most advanced civilization of their time, and had plumbing, three story buildings, urban planning and possibly even primitive optics in 2000 BC (I wrote a bit about them here). Although they've no doubt been romanticized, in the romanticized version at least they had a pretty equitable society, gave women high status, and revered art and nature. Then they were all destroyed by a giant volcano. I remember reading one historian's speculation that if they'd lived, a man would've landed on the moon by 1 AD.

I don't have such antipathy to Christianity that I'd want to prevent it from ever existing, but it sure did give us 2000 odd years of boring religion. Julian the Apostate was a Roman emperor who ruled a few reigns after Constantine and tried to turn back the clock, de-establish Christianity, and revive all the old pagan cults. He was also a philosopher, an intellectual, and by most accounts a pretty honest and decent guy. He died after reigning barely over a year, from a spear wound incurred in battle. If he'd lived, for all we know the US could be One Nation Under Zeus (or Wodin, or whoever) right now.

As for Alexander the Great, he was just plain nifty. I think I heard he was planning a campaign against Carthage before he died. If he'd lived to 80, he could've conquered all Europe, North Africa, and Western Asia, and have unified the whole western world under a dynasty of philosopher-kings dedicated to spreading Greek culture and ideas. Given a few more years, he might also have solved that whole "successor" issue.

Are wireheads happy?

108 Post author: Yvain 01 January 2010 04:41PM

Related to: Utilons vs. Hedons, Would Your Real Preferences Please Stand Up

And I don't mean that question in the semantic "but what is happiness?" sense, or in the deep philosophical "but can anyone not facing struggle and adversity truly be happy?" sense. I mean it in the totally literal sense. Are wireheads having fun?

They look like they are. People and animals connected to wireheading devices get upset when the wireheading is taken away and will do anything to get it back. And it's electricity shot directly into the reward center of the brain. What's not to like?

Only now neuroscientists are starting to recognize a difference between "reward" and "pleasure", or call it "wanting" and "liking". The two are usually closely correlated. You want something, you get it, then you feel happy. The simple principle behind our entire consumer culture. But do neuroscience and our own experience really support that?

It would be too easy to point out times when people want things, get them, and then later realize they weren't so great. That could be a simple case of misunderstanding the object's true utility. What about wanting something, getting it, realizing it's not so great, and then wanting it just as much the next day? Or what about not wanting something, getting it, realizing it makes you very happy, and then continuing not to want it?

The first category, "things you do even though you don't like them very much" sounds like many drug addictions. Smokers may enjoy smoking, and they may want to avoid the physiological signs of withdrawl, but neither of those is enough to explain their reluctance to quit smoking. I don't smoke, but I made the mistake of starting a can of Pringles yesterday. If you asked me my favorite food, there are dozens of things I would say before "Pringles". Right now, and for the vast majority of my life, I feel no desire to go and get Pringles. But once I've had that first chip, my motivation for a second chip goes through the roof, without my subjective assessment of how tasty Pringles are changing one bit.

Think of the second category as "things you procrastinate even though you like them." I used to think procrastination applied only to things you disliked but did anyway. Then I tried to write a novel. I loved writing. Every second I was writing, I was thinking "This is so much fun". And I never got past the second chapter, because I just couldn't motivate myself to sit down and start writing. Other things in this category for me: going on long walks, doing yoga, reading fiction. I can know with near certainty that I will be happier doing X than Y, and still go and do Y.

Neuroscience provides some basis for this. A University of Michigan study analyzed the brains of rats eating a favorite food. They found separate circuits for "wanting" and "liking", and were able to knock out either circuit without affecting the other (it was actually kind of cute - they measured the number of times the rats licked their lips as a proxy for "liking", though of course they had a highly technical rationale behind it). When they knocked out the "liking" system, the rats would eat exactly as much of the food without making any of the satisifed lip-licking expression, and areas of the brain thought to be correlated with pleasure wouldn't show up in the MRI. Knock out "wanting", and the rats seem to enjoy the food as much when they get it but not be especially motivated to seek it out. To quote the science1:

Pleasure and desire circuitry have intimately connected but distinguishable neural substrates. Some investigators believe that the role of the mesolimbic dopamine system is not primarily to encode pleasure, but "wanting" i.e. incentive-motivation. On this analysis, endomorphins and enkephalins - which activate mu and delta opioid receptors most especially in the ventral pallidum - are most directly implicated in pleasure itself. Mesolimbic dopamine, signalling to the ventral pallidum, mediates desire. Thus "dopamine overdrive", whether natural or drug-induced, promotes a sense of urgency and a motivation to engage with the world, whereas direct activation of mu opioid receptors in the ventral pallidum induces emotionally self-sufficient bliss.

The wanting system is activated by dopamine, and the liking system is activated by opioids. There are enough connections between them that there's a big correlation in their activity, but the correlation isn't one and in fact activation of the opioids is less common than the dopamine. Another quote:

It's relatively hard for a brain to generate pleasure, because it needs to activate different opioid sites together to make you like something more. It's easier to activate desire, because a brain has several 'wanting' pathways available for the task. Sometimes a brain will like the rewards it wants. But other times it just wants them.

So you could go through all that trouble to find a black market brain surgeon who'll wirehead you, and you'll end up not even being happy. You'll just really really want to keep the wirehead circuit running.

Problem: large chunks of philosophy and economics are based upon wanting and liking being the same thing.

By definition, if you choose X over Y, then X is a higher utility option than Y. That means utility represents wanting and not liking. But good utilitarians (and, presumably, artificial intelligences) try to maximize utility (or do they?). This correlates contingently with maximizing happiness, but not necessarily. In a worst-case scenario, it might not correlate at all - two possible such scenarios being wireheading and an AI without the appropriate common sense.

Thus the deep and heavy ramifications. A more down-to-earth example came to mind when I was reading something by Steven Landsburg recently (not recommended). I don't have the exact quote, but it was something along the lines of:

According to a recent poll, two out of three New Yorkers say that, given the choice, they would rather live somewhere else. But all of them have the choice, and none of them live anywhere else. A proper summary of the results of this poll would be: two out of three New Yorkers lie on polls.

This summarizes a common strain of thought in economics, the idea of "revealed preferences". People tend to say they like a lot of things, like family or the environment or a friendly workplace. Many of the same people who say these things then go and ignore their families, pollute, and take high-paying but stressful jobs. The traditional economic explanation is that the people's actions reveal their true preferences, and that all the talk about caring about family and the environment is just stuff people say to look good and gain status. If a person works hard to get lots of money, spends it on an iPhone, and doesn't have time for their family, the economist will say that this proves that they value iPhones more than their family, no matter what they may say to the contrary.

The difference between enjoyment and motivation provides an argument that could rescue these people. It may be that a person really does enjoy spending time with their family more than they enjoy their iPhone, but they're more motivated to work and buy iPhones than they are to spend time with their family. If this were true, people's introspective beliefs and public statements about their values would be true as far as it goes, and their tendency to work overtime for an iPhone would be as much a "hijacking" of their "true preferences" as a revelation of them. This accords better with my introspective experience, with happiness research, and with common sense than the alternative.

Not that the two explanations are necessarily entirely contradictory. One could come up with a story about how people are motivated to act selfishly but enjoy acting morally, which allows them to tell others a story about how virtuous they are while still pursuing their own selfish gain.

Go too far toward the liking direction, and you risk something different from wireheading only in that the probe is stuck in a different part of the brain. Go too far in the wanting direction, and you risk people getting lots of shiny stuff they thought they wanted but don't actually enjoy. So which form of good should altruists, governments, FAIs, and other agencies in the helping people business respect?

Sources/Further Reading:

1. Wireheading.com, especially on a particular University of Michigan study

2. New York Times: A Molecule of Motivation, Dopamine Excels at its Task

3. Slate: The Powerful and Mysterious Brain Circuitry...

4. Related journal articles (1, 2, 3)

Comment author: pjeby 30 December 2009 12:15:01AM 5 points [-]

Either "identity" is too vague or I don't understand how you're using it. There's no explanation of what an identity is, why or how people seek an identity, or why they would seek one instead of others.

An "identity" is a label attached to a set of personal attributes that signify membership in a subgroup, e.g. "A Spartan comes back with his shield or on it".

The subgroup can be political, familial, or other: "A Smith never backs down", "A Scout is always prepared", and "Big boys don't cry".

People seek to emulate identities they are attracted to -- i.e., ones with whom they feel they already have something in common, and which offer them something in return. (This latter bit is vague: the something in return could be the admiration of allies or the annoyance of enemies. E.g., being a punk rocker to piss off your parents.)

(And of course, these feelings of attraction aren't any more consciously thought out than sexual attraction is.)

"Village idiot" is an identity and "brilliant seeker of truth" is an identity, but most people, given the choice, would try to conform to the latter.

But not all people. A person whose natural talents are reinforced in that direction will likely end up there... see for example the "class clown".

Human beings tend to be different from one another because reinforcement leads to a positive feedback loop of increasing "talent" (i.e. skill) in being a particular personality type. People then try to "fit in" somewhere, even if the fit is a minority role of one.

"Living up to ideals" is a very human-level thought. Where's the mental circuitry behind it? Why would people want to live up to ideals, or even have ideals? What's my motivation?

I don't understand whether you mean "why" in an evolutionary sense, or "why" in the sense of "what causes it" (i.e. how).

I think you're entirely right about identity, but that identity is a high-level process that emerges out of the search for status. Exactly how is a whole other post, but I think a lot of the research you mention is in the fields of contingencies of self-worth, ie how our self-esteem comes from lots of different sources. We then value or devalue those sources in order to maximize our own self-esteem.

I think it's a mistake to use "status" as a single lump term for all these things. We don't directly perceive our "status" in an absolute sense, and status is in any case relative. I think the emotion that's relevant in this case is the one that some researchers refer to as "elevation" -- the opposite of disgust. We aspire to be like those who inspire us, and we feel pride in having an identity as a worthy member of a subgroup.

This is not the same thing as feeling that we have a high status within a subgroup, or within a larger group. Beware the Big Hammer. ;-)

While "self-esteem" certainly mirrors one's actual status feedback in part, it is not a direct measurement, nor is it exclusively based on status.

Comment author: Yvain 30 December 2009 12:17:30AM 1 point [-]

I think we more or less agree except on semantic issues, then. If I ever manage to continue this sequence, it'll become clearer whether we do or don't.

Comment author: happyseaurchin 29 December 2009 03:11:20PM 0 points [-]

Nice post, thanks.

"In this model, people aren't just seeking status, they're (also? instead?) seeking a state of affairs that allows them to believe they have status."

  1. Replace also or instead with rather. That is, the default state of mind is that individuals believe they have status. This might be through the regular strategy of seeking social wealth (icons, respect, position, possessions), as well as through invisibles (A Big Idea, the truth, The Secret). The status is always self-assigned; think of those who do do not accept the status conferred upon them. Which leads to...

  2. I am specifically interested in individuals who are not playing the status game, who consider themselves end-nodes, nobody's, or self-less. Consider Mother Teresa as an example perhaps. How does your model deal with this? (This line of thinking might parallel altruistic behaviour, which might be a useful space to connect up.)

  3. More needs to be said about relative scale, that is cultures and subcultures, families and tribes. There are multiple superimposed groupings going on in any specific individual's life (daughter, sister, wife, mother, colleague, friend, consumer, etc), and I look forward to reading this.

Comment author: Yvain 29 December 2009 06:51:46PM 2 points [-]

Mother Teresa isn't hard to explain on this model. She could be gaining normal status covertly, like Michael Vassar says. She could be feeling good about herself and her status because she thinks that her altruism makes her a better person than others. Or she could feel good about herself and her status because she's serving God, which makes her a better person than others.

Comment author: MichaelVassar 29 December 2009 06:40:00PM 1 point [-]

Some of my recent tweets hint at a theory of where identity comes from, but a write-up will take some time. I think I agree with you regarding contingencies of self worth and the desire for no feedback.

Comment author: Yvain 29 December 2009 06:44:22PM *  2 points [-]

I look forward to a day when all great philosophical systems can be expressed in 140 or fewer characters (no, really, I just found your Twitter feed and really like it)

Comment author: whpearson 29 December 2009 06:20:52PM *  0 points [-]

Zir or hir. According to wikipedia

I prefer Ve, because that was the first one I came across. I forget which one Eliezer uses, but I have seen him use one, so we are ground breaking in a number of different directions. I'd like to standardise if possible and they (sing) is not sufficient.

Comment author: Yvain 29 December 2009 06:42:31PM 2 points [-]

I always thought ve was limited to transhumans.

Comment author: Oscar_Cunningham 29 December 2009 01:07:04PM *  0 points [-]

"A rat isn't exactly seeking an optimum level of food, it's seeking an optimum ratio of ventromedial to ventrolateral hypothalamic stimulation, or, in rat terms, a nice, well-fed feeling."

So if I move my hand away from a hot pan, am I actually seeking to: "move my hand away from a hot pan" or

"avoid touching the pan" or

"avoid being burnt" or

"avoid pain receptors in my hand being activated" or

"avoid neural signals in my brain that correspond to pain" or

"avoid the feeling of pain"?

Someone needs to do some buck-stopping or else the master-slave model will turn into a master-slave1-slave2-slave3... model. Although come to think of it, that might me more correct. (EDIT: Note to self, line spacing is weird, I'm off to look in the wiki)

Comment author: Yvain 29 December 2009 03:45:14PM 3 points [-]

Moving away from a hot pan is a reflex action. You're not seeking anything, it's done before "you" even enter the picture. But in the general case, it's a combination of consciously avoiding bodily damage (you don't want your skin burnt off), and avoiding pain, which is a correlate of bodily damage.

Avoiding pain is probably the stronger motive, since lower animals who can't think far enough to worry about long-term bodily damage will do the same and since something that causes pain but not damage (that Bene Gesserit box in Dune Paul had to stick his hand in, for example) will cause the same effect.

The buck-stopping problem is a confusion of levels. On the conscious, human level, goal is to minimize pain (the human doesn't even know there's such a thing as pain receptors unless ze knows some neuroscience). On the unconscious inhuman level, "goal" is meaningless, and it would be better to talk about transmitters moving down electrochemical gradients and such.

In response to comment by Yvain on Suffering
Comment author: DanielLC 29 December 2009 03:29:32AM *  1 point [-]

Wouldn't it be enough to have positive /or/ negative qualia?

In response to comment by DanielLC on Suffering
Comment author: Yvain 29 December 2009 01:02:32PM 0 points [-]

...yes.

Comment author: pjeby 29 December 2009 05:05:14AM 19 points [-]

Here's a piece that I think you're missing: identity and status are related, but not equivalent.

Identity is about living up to a social standard or ideal for a role that defines your place in the tribe. Living up to "your" ideals (i.e., the tribe's standard for the role) produces good feelings.

Let's say that a tribe has hunters, gatherers, warriors, shamans, and healers. Each subgroup (subculture?) has a set of practices, sayings, beliefs, values, etc. that are unique to that subgroup role. In order for an individual to occupy a productive specialization, they have to learn (and be motivated to embody) these standards and practices.

Also notice that what's high-status behavior for each subgroup is different; behavior that's honored when done by a shaman would be laughed at (or worse) in a hunter.

Thus, we get the all-too-human phenomena of conforming non-conformists, status-seeking behavior by people who claim that all status is beneath them, etc.

So, I think you're on the right general track, but missing a more specific mechanism that more closely explains why this type of behavior is rewarded. It's not status-seeking per se, it's "living up to ideals". Conspiracy theorists are emulating the ideal of a revolutionary truthseeker... and so, perhaps are most of us here. ;-)

Thing is, it's not the specific behaviors or results that are rewarded by this mechanism; it's attitudes, emotions, and other fuzzy stuff like that. So, you can be a really fuzzy thinker and still pride yourself on being a brilliant seeker of truth... in attitude. (Presumably, in the ancestral environment, your actual skill calibration would occur via real-world feedback and the not-so-gentle correction of your peers or mentors; but the motivation to persist in the learning would come via the pride-of-identity mechanism.)

Priming research, btw, shows that when we're reminded of the subgroups we belong to, our behaviors tend to conform to ideals or stereotypes of those subgroups -- IOW, identity, not status, is the key to stereotypical behavior. (And incidentally, it's a mild refutation of the idea that status needs drive everything. Human beings do have other motivators.)

Comment author: Yvain 29 December 2009 12:54:06PM *  13 points [-]

Either "identity" is too vague or I don't understand how you're using it. There's no explanation of what an identity is, why or how people seek an identity, or why they would seek one instead of others. "Village idiot" is an identity and "brilliant seeker of truth" is an identity, but most people, given the choice, would try to conform to the latter.

"Living up to ideals" is a very human-level thought. Where's the mental circuitry behind it? Why would people want to live up to ideals, or even have ideals? What's my motivation?

I think you're entirely right about identity, but that identity is a high-level process that emerges out of the search for status. Exactly how is a whole other post, but I think a lot of the research you mention is in the fields of contingencies of self-worth, ie how our self-esteem comes from lots of different sources. We then value or devalue those sources in order to maximize our own self-esteem. I'm pretty smart but not too strong, so I come up with a worldview in which intellect is much more important than physical strength, and my identities, like "rationalist" and "leftist with a side of libertarianism" develop partly out of what helps me affirm that story that leads to my high status and high self-worth.

I disagree about the attitude versus results thing. One of the posts I still have to make in this sequence argues that this mechanism is what drives so many people into roles that can't receive feedback. For example, you won't find many poor people priding themselves on how rich they are, or too many stupid people priding themselves on how book-smart and well-educated they are, but anyone can pride themselves on how moral they are and how correct their political beliefs are, and most people do. Likewise, the 9-11 Truther example and other conspiracies of fact tend to form around questions that are hard to resolve.

Also, although you use the example of "shaman", there weren't that many roles in the EEA, shamans are probably a pretty late development (first ceremonial burial isn't until 100,000 BC or so), and everything else came even later.

Summary: I think you're right about roles and identity, but the goal of this post is to deconstruct "identity" into moving parts.

Comment author: Alicorn 29 December 2009 03:10:33AM 0 points [-]

and wouldn't get away with me get away with it?

I suspect that this is supposed to be "and wouldn't let me get away with it?"

Comment author: Yvain 29 December 2009 03:30:19AM *  0 points [-]

Ah, the perils of partial editing. Thanks and fixed.

That other kind of status

72 Post author: Yvain 29 December 2009 02:45AM

"Human nature 101.  Once they've staked their identity on being part of the defiant elect who know the Hidden Truth, there's no way it'll occur to them that they're our catspaws." - Mysterious Conspirator A

This sentence sums up a very large category of human experience and motivation. Informally we talk about this all the time; formally it usually gets ignored in favor of a simple ladder model of status.

In the ladder model, status is a one-dimensional line from low to high. Every person occupies a certain rung on the ladder determined by other people's respect. When people take status-seeking actions, their goal is to to change other people's opinions of themselves and move up the ladder.

But many, maybe most human actions are counterproductive at moving up the status ladder. 9-11 Conspiracy Theories are a case in point. They're a quick and easy way to have most of society think you're stupid and crazy. So is serious interest in the paranormal or any extremist political or religious belief. So why do these stay popular?

Could these just be the conclusions reached by honest (but presumably mistaken) truth-seekers unmotivated by status? It's possible, but many people not only hold these beliefs, but flaunt them out of proportion to any good they could do. And there are also cases of people pursuing low-status roles where there is no "fact of the matter". People take great efforts to identify themselves as Goths or Juggalos or whatever even when it's a quick status hit.

Classically people in these subcultures are low status in normal society. Since subcultures are smaller and use different criteria for high status, maybe they just want to be a big fish in a small pond, or rule in Hell rather than serve in Heaven, or be first in a village instead of second in Rome. The sheer number of idioms for the idea in the English language suggests that somebody somewhere must have thought along those lines.

But sometimes it's a subculture of one. That Time Cube guy, for example. He's not in it to gain cred with all the other Time Cube guys. And there are 9-11 Truthers who don't know any other Truthers in real life and may not even correspond with others online besides reading a few websites.

Which brings us back to Eliezer's explanation: the Truthers have "staked their identity on being part of the defiant elect who know the Hidden Truth". But what does that mean?

A biologist can make a rat feel full by stimulating its ventromedial hypothalamus. Such a rat will have no interest in food even if it hasn't eaten for days and its organs are all wasting away from starvation. But stimulate the ventrolateral hypothalamus, and the rat will feel famished and eat everything in sight, even if it's full to bursting. A rat isn't exactly seeking an optimum level of food, it's seeking an optimum ratio of ventromedial to ventrolateral hypothalamic stimulation, or, in rat terms, a nice, well-fed feeling.

And humans aren't seeking status per se, we're seeking a certain pattern of brain activation that corresponds to a self-assessment of having high status (possibly increased levels of dopamine in the limbic system). In human terms, this is something like self-esteem. This equation of self esteem with internal measurement of social status is a summary of sociometer theory.

So already, we see a way in which overestimating status might be a very primitive form of wireheading. Having high status makes you feel good. Not having high status, but thinking you do, also makes you feel good. One would expect evolution to put a brake on this sort of behavior, and it does, but there may be an evolutionary incentive not to arrest it completely.

If self esteem is really a measuring tool, it is a biased one. Ability to convince others you are high status gains you a selective advantage, and the easiest way to convince others of something is to believe it yourself. So there is pressure to adjust the sociometer a bit upward.

So a person trying to estimate zir social status must balance two conflicting goals. First, ze must try to get as accurate an assessment of status as possible in order to plan a social life and predict others' reactions. Second, ze must construct a narrative that allows them to present zir social status as as high as possible, in order to reap the benefits of appearing high status.

The corresponding mind model1 looks a lot like an apologist and a revolutionary2: one drive working to convince you you're great (and fitting all data to that theory), and another acting as a brake and making sure you don't depart so far from reality that people start laughing.

In this model, people aren't just seeking status, they're (also? instead?) seeking a state of affairs that allows them to believe they have status. Genuinely having high status lets them assign themselves high status, but so do lots of other things. Being a 9-11 Truther works for exactly the reason mentioned in the original quote: they've figured out a deep and important secret that the rest of the world is too complacent to realize.

It explains a lot. Maybe too much. A model that can explain anything explains nothing. I'm not a 9-11 Truther. Why not? Because my reality-brake is too strong, and it wouldn't let me get away with it? Because I compensate by gaining status from telling myself how smart I am for not being a gullible fool like those Truthers are? Both explanations accord with my introspective experience, but at this level they do permit a certain level of mixing and matching that could explain any person holding or not holding any opinion.

In future posts in this sequence, I'll try to present some more specifics, especially with regard to the behavior of contrarians.

 

Footnotes

1. I wrote this before reading Wei Dai's interesting post on the master-slave model, but it seems to have implications for this sort of question.

2. One point that weakly supports this model: schizophrenics and other people who lose touch with reality sometimes suffer so-called delusions of grandeur. When the mind becomes detached from reality (loses its 'brake'), it is free to assign itself as high a status as it can imagine, and ends up assuming it's Napoleon or Jesus or something like that.

Comment author: Yvain 25 December 2009 01:09:36AM *  9 points [-]

Whether rationality is or isn't the best way to get things done depends very much on what those things are. Talk of generic success is either meaningless or a covert importation of cultural baggage.

Linguistic qualms aside, if you're cautioning people against expecting rationality to be an quick road to material wealth and power then I completely agree.

In response to Two Truths and a Lie
Comment author: Yvain 23 December 2009 01:31:46PM *  24 points [-]

It's an interesting experiment, and probably a good teaching exercise under controlled conditions to teach people about falsificationism, but real theories are too complex and theories about human behavior are way too complex.

Take the "slam dunk" theory of evolution. If "Some people and animals are homosexual" was in there, I'd pick that as the lie without even looking at the other two (well, if I didn't already know). There are some okay explanations of how homosexuality might fit into evolution, but they're not the sort of thing most people would start thinking about unless they already knew homosexuality existed.

(Another example: plate tectonics and "Hawaii, right smack in the middle of a huge plate, is full of volcanoes".)

In response to comment by LauraABJ on Karma Changes
Comment author: Eliezer_Yudkowsky 22 December 2009 03:23:15AM 11 points [-]

Well... sorry to put it this way, but when you said that, I checked to see if you'd posted anything, and you hadn't. There's a rule I have which often offends people, and yet it seems like a very important rule, which is when someone tells you that change X will prevent people from doing Y, and they aren't doing Y, you probably want to check with people who are currently doing Y about that.

Comment author: Yvain 23 December 2009 12:29:52AM *  21 points [-]

Take as a sample some of the people who have made nonmeta top-level posts in the past week or so and aren't super-regular posters. I count Shalmanese, David Balan, Matt, Mr. Hen, Warrigal, and JHuffman. A few of the posts I downvoted, but none was so abominably stupid that the person involved should be ridden out of town on a rail.

I looked up how long it took each of those posters to earn their 20 karma/50 karma based on comment points alone (not counting comments replying to their own posts). IE, how far do you have to go back for all comments between then and now to total >20/50 karma? It was quick and involved a lot of skimming and mental math, but it was something like:

  • Matt: 20 September, 50 May
  • JHuffman: 20 April, 50 March
  • MrHen: 20 July, 50 June
  • Warrigal: 20 early Dec, 50 early Nov
  • Shalmanese: 20 September, not at 50 yet
  • David Balan: AFAICT only comments on his own posts

If these numbers are representative of people who aren't super-regulars but like making top-level posts, they're getting comment karma on other people's posts somewhere like five to ten per month.

That means it takes someone with their usage habits 2-4 months to get 20 comment karma and 5-10 months to get 50. It also means that if one of these people's first posts made a dumb mistake and got them voted down to -5, then at 50 point threshold it'd take another 5-10 months for them to be able to make their second.

The cost of having too low a threshold is that there are a few dumb posts that never get promoted to where anyone can see them anyway. The cost of having too high a threshold is that we miss out on interesting information and the community doesn't expand.

Lower the posting threshold back to 20, and maybe do something about first post downvotes.

In response to Karma Changes
Comment author: Yvain 22 December 2009 02:23:12AM *  4 points [-]

Why change minimum karma to start posting?

In response to comment by Yvain on The Sacred Mundane
Comment author: mattnewport 18 December 2009 11:01:29PM 7 points [-]

I'm a little skeptical of this claim. When I've experienced sleep paralysis I've imagined seeing a non-supernatural human intruder but all I actually saw was a vaguely human shaped shadow which for some reason in the confused half-asleep state of sleep paralysis seems highly likely to be an ill-intentioned intruder rather than a shadow. People with a different cultural expectation might claim to have 'seen' a demon but I don't think that should necessarily be interpreted as them having had a detailed hallucination, just that an ambiguous and threatening presence is assumed to be whatever strikes them as the most likely thing to be hanging around threateningly if indistinctly.

Comment author: Yvain 21 December 2009 02:26:33AM 1 point [-]

Never having had sleep paralysis, I bow to your superior expertise on the subject.

Comment author: Yvain 20 December 2009 11:10:44PM *  3 points [-]

Interesting, but I'd like to see you expand on exactly the rationality mechanisms by which people get "tricked" by these contracts, and what kind of rules you might use to classify a contract as "trickery" versus "I wouldn't take it, but another reasonable person might."

I don't see a qualitative difference between a case like "Let's ban outrageous interest on credit card loans because rational people wouldn't accept this", "Let's ban certain drugs that we're not sure work because rational people wouldn't use them," and "Let's ban jobs without a certain minimum wage and safety regulations because rational people wouldn't take them". Right now, I judge these cases on a very intuitive "I'll know an irrational contract when I hear it" sort of basis, but the libertarian method of rejecting all of them alike attracts me with its elegance. Does research in this area give any pointers to what standards we would use to discriminate?

In response to comment by Yvain on Rationalist Fiction
Comment author: Bindbreaker 11 December 2009 06:26:42AM *  3 points [-]

Started reading the first one-- from the prologue alone, Kellhus seems absurdly strong/skilled/fast. He reads people's minds by looking at the patterns of their facial muscles, catches arrows out of the air, kills large groups of enemies by himself in hand-to-hand combat, etc. I'm not sure what lessons could really be derived from this, since these actions are far beyond the realm of normal human ability. Does the series/book get any better, or am I missing something here?

Comment author: Yvain 18 December 2009 10:54:12PM 1 point [-]

I've heard this complaint from others, and it's valid. Where the series really starts coming into its own, in my opinion, is around the end of the first book/ start of the second where Kellhus gets involved in politics and persuasion. This is the part that gives me a better understanding of "superintelligences" and what they might do.

In response to comment by Yvain on The Sacred Mundane
Comment author: Eliezer_Yudkowsky 16 December 2009 10:42:43PM 5 points [-]

This notion of "dhyana experience" as completely unconditioned sounds suspiciously modernized-religious to me. According to the sadly-former-atheist John C. Wright, when he gets these hugely powerful "religious experiences", he gets the Trinity - yes, the good 'ol fashioned Trinity - talking to him directly.

Comment author: Yvain 18 December 2009 10:51:40PM 10 points [-]

From above:

This isn't interpreted as a post-hoc attribution; just as the paranoid feels like it's the CIA after them, the Christian feels like they just saw Jesus.

Another example: in sleep paralysis, many people report seeing demonic type figures. Although I haven't been able to find any explicit evidence, I've seen suggestions that the exact variety of demon depends on the sleeper's expectation. For example, Chinese see something like a classic transparent ghost, Hmong see a tiny child-like figure, and Americans see stuff like typical horns-and-tail demons or typical pointy-hat type witches.

The mental "stimulus" in sleep paralysis doesn't have any features - it's just a general feeling of fear, unreality, and oppression. But the sufferer does see a demon or monster with the culturally appropriate features.

So it's not contradictory to say both that dhyana itself is an "unconditioned" experience, and that individual experiences of dhyana can be detailed - although there may be many different types of emotionally powerful hallucination and "unconditioned" may be too vague to be a useful word.

In response to comment by taw on Rebasing Ethics
Comment author: Sebastian_Hagen 15 December 2009 03:01:37PM *  1 point [-]

You could certainly make a consistent argument along those lines. To the extent that waiters get consistent tips, this should lead to either one of two outcomes:

  1. Their employers will pay them less, correctly reasoning that since their effective income is boosted from an external source, the employer can pay them a lower nominal wage and still attract the same quality of employees.
  2. If 1. doesn't happen for whatever reason (e.g. because they're already at the minimum wage), this will effectively push the waiter job into a higher pay grade, leading to job gentrification (i.e. restaurants will hire more competent and expensive employees, and the people currently doing waiting will no longer be qualified for the job).

Tipping might make sense if you did it selectively - if you tipped people proportionally to the quality of the service they gave you personally, and made sure the tip doesn't exceed the gains you got through the better-than-baseline waiting. That would motivate them to produce more positive-sum gains while waiting you, and actually make society better off. But the trick here is the selective rewards, not the tipping.

Comment author: Yvain 15 December 2009 06:39:25PM 3 points [-]

I'd predict that restaurants would start charging 15% more on all meals. Waiters would still get about the same amount. The only difference is that the money would be taken equally from generous and stingy people instead of coming disproportionately from the generous, and given equally to good and bad waiters instead of going disproportionately to the good.

In response to comment by Yvain on Rebasing Ethics
Comment author: Shalmanese 15 December 2009 03:56:27PM 1 point [-]

I deliberately chose an innocuous example so as to not overly trip the discussion into the specifics of the example itself. I'm not going to talk about some of the more extreme examples of what this would imply until other people do.

You're correct in that modifying tipping behavior by itself would probably not be worth being a dick about in the same way that just switching to low fat milk is probably not worth absorbing all of our science of nutrition & dieting about. You have to be able to see the cumulative effects of a complete rebasing before you can judge it's ultimate utility.

As for whether it's worth it, I think you need to look at where a person wants to be vs where they actually are. Looking out in the world, I don't see a lot of rationalists of the type who inhabit this board who are rich, powerful, admired, have happy marriages or have fulfilled the potential they believe they have. I'm not promising that you'll have all of that if you just rebase your ethics but If you happy not to try just so you can keep your warm fuzzy moral feelings, that's, of course, your own choice.

In response to comment by Shalmanese on Rebasing Ethics
Comment author: Yvain 15 December 2009 06:34:25PM *  7 points [-]

That helps me understand where the evolution reference comes from and how the tip fits in, but I still don't understand where you get your preference.

Most people would like to be richer, more powerful, and more admired, and it may be that their morality is keeping them from that. But most people would probably also like to be more moral, more compassionate, and contribute more to society, and their desire for material success is keeping them from that. Moral choices are a trade-off between these two desires.

Like all trade-offs, which option to choose depends on the value of each good involved. In a choice between getting an extra dollar and saving a million lives, I'd choose the lives. In a choice between getting a million dollars and preventing one other person getting a dust-speck-in-the-eye, I'd take the million. The question isn't whether to take moral or material goods, it's at what rate to exchange them.

This post seems to be arguing that people consistently overvalue moral goods and undervalue material goods. But when psychologists actually study the issue, they find the opposite: that moral goods are much more effective at purchasing life satisfaction and personal happiness than material goods (This study is the first I found, and not necessarily the best, of a large number).

If increasing our consumption of material as opposed to moral goods isn't justified by evolutionary history and doesn't make us happier, what exactly is the advantage?

In response to Rebasing Ethics
Comment author: Yvain 15 December 2009 02:42:24PM *  11 points [-]

Altruism is no longer valuable in evolutionary terms, but who cares?.

What's important to us as people isn't evolutionary value, it's satisfying personal preferences derived causally but not normatively from evolutionary value. Having money is one of those preferences. Being a nice person is another one of those preferences. If you deliberately make a choice that satisfies weaker preferences than the alternative, you're just throwing away utility for no reason.

If you haven't already, I suggest reading the Evolution Sequence, especially Alien God, Adaption-Executors, and Evolutionary Psychology, and the Metaethics Sequence

Now, if you want to make this into a reaaaallly repulsive ethical dilemma, add that true utilitarians should refuse to tip so they can donate that money to a charity that produces greater utility than the tip does.

Comment author: wedrifid 13 December 2009 05:26:48AM 5 points [-]

How did Less Wrong do by comparison? The average estimated probability of Amanda Knox's guilt was 0.35 (thanks to Yvain for doing the calculation).

Yvain! How could you? What did the probabilities do to deserve that kind of abuse? (I strongly assert that averaging the probabilities is not a good way to combine such estimates.)

Comment author: Yvain 13 December 2009 03:01:12PM 3 points [-]

...and why do you assert that? If you have good reasons, I'd like to see a top level post on the subject, since this is my natural response to a bunch of probability estimates given by different people with the same information who are rational enough that I care what they think.

Comment author: Yvain 13 December 2009 02:36:02PM 9 points [-]

No different from the prior, which is dominated by the probability that someone in whatever reference class you would have put Amanda into on January 1, 2007 would commit murder within twelve months. Something on the order of 0.001 at most.

Out of one thousand criminal trials in which the Less Wrong conventional wisdom gave the defendant a 35% chance of being guilty, you would expect to be able to correctly determine guilt nine hundred ninety nine times?

Comment author: Yvain 13 December 2009 02:04:30PM *  6 points [-]

Let him who has never used time in a less than maximally-utility-producing way cast the first stone.

Comment author: Nick_Tarleton 11 December 2009 04:13:13PM 2 points [-]

giving every possible arrangement of objects/atoms/information equal weight

Without an arbitrary upper bound on complexity, there are infinitely many possible arrangements.

Comment author: Yvain 12 December 2009 05:37:56PM 3 points [-]

Theoretically, it's not infinite because of the granularity of time/space, speed of light, and so on.

Practically, we can get around this because we only care about a tiny fraction of the possible variation in arrangements of the universe. In a coin flip, we only care about whether a coin is heads-up or tails-up, not the energy state of every subatomic particle in the coin.

This matters in the case of a biased coin - let's say biased towards heads 66%. This, I think, is what Wei meant when he said we couldn't just give equal weights to all possible universes - the ones where the coin lands on heads and the ones where it lands on tails. But I think "universes where the coin lands on heads" and "universes where the coin lands on tails" are unnatural categories.

Consider how the probability of winning the lottery isn't .5 because we choose with equal weight between the two alternatives"I win" and "I don't win". Those are unnatural categories, and instead we need to choose with equal weight between "I win", "John Q. Smith of Little Rock Arkansas wins", "Mary Brown of San Antonio, Texas, wins" and so on to millions of other people. The unnatural category "I don't win" contains millions of more natural categories.

So on the biased coin flip, the categories "the coin lands heads" and "the coin lands tails" contains a bunch of categories of lower-level events about collisions of air molecules and coin molecules and amounts of force one can use to flip a coin, and two-thirds of those events are in the "coin lands heads" category. But among those lower-level events, you choose with equal weight.

True, beneath these lower-level categories about collisions of air molecules, there are probably even lower things like vibrations of superstrings or bits in the world-simulation or whatever the lowest level of reality is, but as long as these behave mathematically I don't see why they prevent us from basing a theory of probability on the effects of low level conditions.

Comment author: CarlShulman 11 December 2009 03:03:31PM 1 point [-]

What if they're dead?

Comment author: Yvain 11 December 2009 03:06:22PM 18 points [-]

Well, then I lose the bet...unless someone contacts their ghosts...in which case I win the bet!

Comment author: MichaelVassar 10 December 2009 07:20:33PM 10 points [-]

One problem with this argument is that if psi exists, we are very bad at using it, and we don't see other organisms using it well either. The world we see appears to be almost completely described by normal physics at worst.

I don't think that I'm double-counting evidence. I certainly know that there can be intelligent believers, after all, MANY intelligent people believe that one is compelled to accept the conclusions of the scientific method over those of the scientific community. Also, beliefs can be compelling for any variety of irrational reasons. The evidence I have seen though looks to me like exactly the evidence you would expect given known psychology and no psi. We can surely agree that there is a LOT of evidence that hyman psychology would create belief in psi in the absence of psi, can't we.

I would set my odds at "top twenty most astoundingly surprising things ever discovered but maybe not top ten". That seems to me like odds of many billions to one against, but not trillions. Unfortunately, the odds for almost any plausible winning conditions occurring without psi being real are much higher, making the bet difficult to judge. I have a standing 10,000 to one bet against Blacklight Power's "Hydrino Theory" with Brian Wang based on a personal estimate of odds MUCH less than 1-in-10K for "Hydrino Theory" and I'm happy to extend those odds when the odds are still more favorable, but psychotic breaks by two people in a group of three? If the odds per person are 1%, that gives odds of about 1:3300. I'm happy to give those odds on the Dawkins, Randi Yudkowsky bet and count "psi is actually real" as a rounding error.

Comment author: Yvain 11 December 2009 02:39:15PM 11 points [-]

Have donated $10 to SIAI (seemed less likely to lose you guys money in transaction fees than $1) with public comment about the bet . Will decide where you can donate your $33000 in the unlikely event it proves necessary.

Comment author: Yvain 11 December 2009 02:31:34PM 1 point [-]

Before I’ve observed anything, there seems to be no reason to believe that I’m more likely to be in one world than another, but we can’t let all their weights be equal.

We can't? Why not? Estimating the probability of two heads on two coinflips as 25% is giving existence in worlds with heads-heads, heads-tails, tails-heads, and tails-tails equal weight. The same is true of a more complicated proposition like "There is a low probability that Bigfoot exists" - giving every possible arrangement of objects/atoms/information equal weight, and then ruling out the ones that don't result in the evidence we've observed, few of these worlds contain Bigfoot.

Comment author: Yvain 11 December 2009 04:58:00AM *  7 points [-]

I averaged up the answers given so far in this thread for Knox and got 35% mean, 20% median.

Comment author: Yvain 09 December 2009 11:37:21PM *  3 points [-]

For Knox, Sollecito, and Guede in order: 30, 30, 95, but didn't find either website or Wikipedia enough to feel like I had sufficient information. I think you probably thought the same.

EDIT: After looking at everyone's comments, I'm revisiting to 20, 20, 95. Anyone else want to edit their comments to say how they updated in light of everyone else's opinions?

Comment author: MichaelVassar 08 December 2009 03:38:08AM 16 points [-]

I used to think that way before I knew about Bayesianism. Once I learned about it I realized that the prior probability for psi was very VERY low, e.g. its complex and there's no reason to expect it so one in a bajillion, while the probability for the observed evidence for psi, given what we know about psychology, was well in excess of 50% in the absence of psi, so the update couldn't justify odds greater than two in a bajillion.

Comment author: Yvain 09 December 2009 06:50:16PM *  8 points [-]

You're right that I completely missed the Bayesian boat, and I'm going to have to start thinking more before I speak and revise my estimates down to <1%.

But I'm still reluctant to put them as low as you seem to. The anthropic principle combined with large universe says that whatever complexity is necessary for the existence of conscious observers, we can expect to find at least that level of complexity. Questions like consciousness, qualia, and personal identity still haven't been resolved, and although past experience suggests there is probably a rational explanation to this question, it isn't nearly dissolved yet. If consciousness really is impossible without some exotic consciousness-related physics (Penrosean or otherwise), then our universe will have exotic consciousness-related physics no matter how complex they need to be. And since evolved beings have been so proficient at making use of normal physics to gain sensory information, it's a good bet they'd do the same with exotic consciousness-related physics too if they had them...

...is a somewhat hokey argument I just invented on the spot, and I'm sorry for it. But the ease with which I can put something like that together is itself evidence that there are enough possible sides of the issue that hadn't been considered (at least I hadn't considered that one; maybe you've been thinking about it for years) that it needs at least a little more room for error than two in a bajillion (sorry, Alex).

I also disagree with your assessment of the amount of evidence. Have you ever read any good books by intelligent believers in the subject? It's not all John Edwards psychic chat shows. I also think you might be double-counting evidence against psi here - psi doesn't exist so we know any apparent evidence must come from human psychology, therefore there never was any apparent evidence in the first place. Or have you read the studies and developed separate explanations for each positive result?

Anyway, let's settle this the LW way. Give me your odds that psi exists, and we can make a bet at them. If it's one in a million, then I'll give a cent to your favorite charity on the condition that you give $10,000 to my favorite charity if psi's shown to exist within our lifetimes (defined however you want; possibly as evidence sufficient to convince any two among Randi, Dawkins, and Eliezer that psi is >50% likely).

Comment author: Blueberry 06 December 2009 08:34:14AM *  5 points [-]

You are going so far as to say that there is no possible way that there are hypotheses which have yet to be described which could be understood through the methodology of this particular subgroup. This exercise seems to me to be rejecting these studies intuitively,(without study) just from the ad hominem approach to rejection - well they are parapsychologists therefore they are wrong. If they are wrong, then proper analysis would indicate that, would it not?

This is exactly the point. Parapsychology is one of the very few things we can reject intuitively, because we understand the world well enough to know that psychic powers just can't exist. We can reject them even when proper analysis doesn't indicate that they're wrong, which tells us something about the limitations of analysis.

ETA: Essentially, if the scientific method can't reject parapsychology, that means the scientific method isn't strong enough, not that parapsychology might be legitimate.

Comment author: Yvain 07 December 2009 04:46:17PM *  17 points [-]

There are many other things that people have claimed can be rejected intuitively without study through the years.

In the 18th century, everyone knew that real scientific physics only permitted a body to act upon another body through direct contact. When Newton proposed his theory of gravity, many people rejected it as pseudoscientific or magical because it claimed the stars and planets could exert action at a distance, without saying how they did it.

In the 19th century, everyone knew that life was on a different order than mere matter, because obviously you couldn't produce the self-moving and self-regenerating qualities of life with just stuff like you get in rocks and sand.

In the 20th century, everyone knew that the mind was more than just the brain, since simple introspection could determine the existence of a consciousness inexplicable in simple material terms.

The absurdity heuristic is an okay heuristic, but I'd be really really careful before saying something is so absurd we can throw away any contradictory experimental evidence without a glance.

The possibility I give to some sort of psi effect existing (in a nice, scientific way that we can study once we figure out what form of matter/energy forms its substrate) is pretty low, but not zero. I'm not even willing to give it a tiny one in a bajillion probability - remember that people who say they're 99% sure of something are wrong 20% of the time, and that since this issue is "politically" charged, vaguely defined, and possibly affected by knowledge we don't have, this is exactly the sort of thing we'd be likely to be overconfident on. If this was some calibration test, I wouldn't feel too good about placing more than 95% or so on the nonexistence of psi.

And if you're a Bayesian, a couple of good studies should be able to start manipulating that 5% number upwards

Comment author: Morendil 06 December 2009 09:42:45PM 2 points [-]

That doesn't quite work: how would you keep that ratio low ? In practice, the only way is by countering social influences which might lead a scientist astray with other social influences. The total amount of "social" stays roughly the same.

Consider the LHC, or any particle accelerator. It takes a good deal of "social influence" to get it built, compared to an infinitesimal fraction of its total mass for what scientists hope to observe.

To a very good approximation, any given quark exerts the same influence on a "bad" scientists as it does on a "good" scientist. It takes exceptional and patient work to set up circumstances where the behaviour of a quark, through a long chain of mediating physical influences, results in noticeably different behaviour for a particular scientist.

Generally, there is an enormous amount of "leveraging", for lack of a better word, that needs to happen between some relevant bit of reality under scrutiny at one end, and the kind of scientific consensus on the other end which affords building something like the LHC.

If you wish to study these leveraging effects accurately, you must adopt a symmetrical stance; you have to bear down and study precisely the nature of these enormously long chains of mediation that bridge the gap between reality and our knowledge of it.

Latour for instance does a great job of this kind of description. Pickering's study of Morpurgo in the case of quarks is interesting; I got the sense that Morpurgo is a perfectly good scientist, he just failed to discover quarks. This doesn't jibe with the asymmetric account. I have yet to read "Leviathan and the Air-Pump" which I understand is the original inspiration for the symmetric approach, but apparently Shapin and Schaffer trace these issues all the way back to the debate between Hobbes and Boyle.

This kind of approach gives you a sense of the reality of science as opposed to its mythology - which is largely a product of scientists themselves, for reasons which Latour also outlines convincingly.

It's a messier, more complicated story than the myth - but then reality always is.

Comment author: Yvain 07 December 2009 12:08:45AM *  2 points [-]

That doesn't quite work: how would you keep that ratio low ? In practice, the only way is by countering social influences which might lead a scientist astray with other social influences. The total amount of "social" stays roughly the same.

Well, depends if you want to define "desire to find truth" as a social force. A scientist motivated by a desire to find the truth is a better scientist and more likely to get an accurate result than a scientist motivated by a desire to confirm the tenets of zir religion or political system, or to fit in, or to get a promotion, or to get home early, or any of those other social forces.

The stronger the motivation to find the truth, the less we would expect other, more traditionally "social" forces to influence a scientist, and the more likely that the scientist's results would be accurate.

Because the direction of the motivation to find truth varies along with the evidence, seems fair to say the scientist motivated primarily by truth-seeking is influenced by the evidence and not by the social situation ze's in.

There may not be any human motivated entirely by truth seeking (except of course Eliezer pbuh), but some people are more than others, and that makes those scientists better.

Comment author: Morendil 06 December 2009 06:09:52PM *  2 points [-]

Science is practiced by people, therefore our knowledge about how people act, in particular how they act in situations of interdependence, is directly applicable to scientists.

Thus, I don't find it surprising at all that when we ask the question, "what did we learn about reality and when", the answers include both references to the truth (or reality-correspondence) of scientific facts, and references to the social construction of these very same facts.

Some sociologists of science once came up with an astute observation about historians of science: their accounts exhibited an interesting asymmetry. Whenever a scientist was vindicated, his work would be accounted for on the basis of correspondence with reality. Whenever a scientist was proved wrong, his mistakes would be accounted for on the basis of "social forces" at work.

This asymmetry can only be an artefact of reconstruction after the fact: before a scientic fact has become "knowledge", while it is still in controversy, both reality and social forces are at work on all scientists working on the issue. In fact, "social forces" are merely a name for some aspects of "reality".

Pickering, Latour and others are saying that if the process of science is itself to become an object of knowledge, we need a symmetric account of it, not one which has The Scientist somehow immune to social forces, immune to bias, immune to reality.

That strikes me as entirely unobjectionable.

EDIT: removed "equally" per Yvain's feedback - I just meant to stress that you can't a priori distinguish a good from a bad scientist, they're subject to roughly comparable sets of forces - the word implies stronger symmetry than that, but I don't really need it.

Comment author: Yvain 06 December 2009 08:41:23PM *  5 points [-]

Both reality and social forces are equally at work on all scientists working on the issue

You lost me at "equally" and "all".

Why not just say that both social forces and the part of the natural world under study influence a scientist's decisions, and a scientist becomes a good scientist who draws correct conclusions about the natural world only when ze keeps the ratio of social influence to natural world influence low?

This leads naturally to the conclusion that yes, a disproportionate amount of correct science will be the result of correspondence with reality, and a disproportionate amount of incorrect science will be the result of social forces.

In response to comment by Yvain on Friedman on Utility
Comment author: AndrewKemendo 23 November 2009 02:04:50AM 1 point [-]

it just takes the understanding that five lives are, all things being equal, more important than four lives.

Your examples rely too heavily on "intuitively right" and ceteris paribus conditioning. It is not always the case that five are more important than four and the mere idea has been debunked several times.

if people agree to judge actions by how well they turn out general human preference

What is the method you use to determine how things will turn out?

similarity can probably make them agree on the best action even without complete agreement on a rigorous definition of "well"

Does consensus make decisions correct?

Comment author: Yvain 23 November 2009 02:29:37PM *  9 points [-]

You know the Nirvana fallacy and the fallacy of needing infinite certainty before accepting something as probably true? How the solution is to accept that a claim with 75% probability is pretty likely to be true, and that if you need to make a choice, you should choose based on the 75% claim rather than the alternative? You know how if you refuse to accept the 75% claim because you're virtuously "waiting for more evidence", you'll very likely end up just accepting a claim with even less evidence that you're personally biased towards?

Morality works the same way. Even if you can't prove that one situation will always have higher utility than another, you've still got to go on the balance of probabilities, because that's all you've got.

The last time I used consequentialism in a moral discussion was (thinks back) on health care. I was arguing that when you have limited health care resources, it's sometimes okay to deny care to a "hopeless" case if it can be proven the resources that would be spent on that care could be used to save more people later. So you may refuse to treat one person with a "hopeless" disease that costs $500,000 to treat in order to be able to treat ten people with diseases that cost $50,000.

Now, yes, one of the people involved could be a utility monster. One of the people involved could grow up to be Hitler, or Gandhi, or Ray Kurzweil. Everyone in the example might really be a brain in a vat, or a p-zombie, or Omega, or an Ebborian with constantly splitting quantum mind-sheets. But if you were an actual health care administrator in an actual hospital, would you take the decision that probably fails to save one person, or the decision that probably saves ten people? Or would you say "I have no evidence to make the decision either way", wash your hands of it, and flip a coin?

In this case, it doesn't matter how you define utility; for any person who prefers life to death, there's only one way to proceed. Yet there are many people in the real world, both hospital administrators and especially voters, who would support the other decision - the one where we give one person useless care now but let ten potentially curable people die later - with all their hearts. Our first job is to spread enough consequentialism to get people to stop doing this sort of thing. After that, we can argue about the technical details all we want. We can stop shooting ourselves in the foot even before we have a complete theory of ballistics.

In response to Friedman on Utility
Comment author: Yvain 23 November 2009 12:34:22AM *  6 points [-]

The examples given in the article are bad examples - any decent concept of utility could deal with them pretty easily - but there are good examples he could've used that really do show some underlying ambiguity in the concept around the edges. I think most of those are solveable with enough creativity and enough willingness not to go "Oh, look, something that appears to be a minor surface-level problem, let's immediately give up and throw out the whole edifice!".

But that sort of thing doesn't really matter as regards whether we should use utility for moral judgments. It doesn't have to be perfect, it just has to be good enough. It doesn't take any kind of complicated distinction between hedonism and preference to solve the trolley problem, it just takes the understanding that five lives are, all things being equal, more important than four lives.

This sort of thing is one reason I've tried to stop using the word "utilitarianism" and started using the word "consequentialism". It doesn't set off the same defenses as "utility", and if people agree to judge actions by how well they turn out general human preference similarity can probably make them agree on the best action even without complete agreement on a rigorous definition of "well".

Comment author: Yvain 11 November 2009 09:54:49AM 3 points [-]

New study shows that one of LW's favorite factoids (having children decreases your happiness rather than increases it) may be either false or at least more complex than previously believed: http://blog.newsweek.com/blogs/nurtureshock/archive/2009/11/03/can-happiness-and-parenting-coexist.aspx

Comment author: CronoDAS 11 November 2009 09:49:04AM 1 point [-]

Just a bit of silliness:

With apologies to Brad DeLong, when reading WSJ editorials you need to bear two things in mind:

  1. The WSJ editorial page is wrong about everything.
  2. If you think the WSJ editorial page is right about something, see rule #1.

After all, here’s what you would have believed if you listened to that page over the years: Clinton’s tax hike will destroy the economy, you really should check out those people suggesting that Clinton was a drug smuggler, Dow 36000, the Bush tax cuts will bring surging prosperity, Saddam is backing Al Qaeda and has WMD, there isn’t any housing bubble, US households have a high savings rate if you measure it right. I’m sure I missed another couple of dozen high points.

Reversed stupidity might not be intelligence, but what about reversed malice?

Comment author: Yvain 11 November 2009 09:53:12AM 3 points [-]

Force anyone to express several controversial opinions per day for several decades and you'll be able to cherry pick a list of seven hilariously wrong examples.

Comment author: Bindbreaker 02 November 2009 11:52:19AM 3 points [-]

I've been trying to ease some friends into basic rationality materials but am running into a few obstacles. Is there a quick and dirty way to deal with the "but I don't want to be rational" argument without seeming like Mr. Spock? Also, what's a good source on the rational use of emotions?

Comment author: Yvain 02 November 2009 03:00:21PM *  7 points [-]

To Eliezer's list, I would add "Something To Protect" and the very end of "Circular Altruism". When a friend of mine said something similar during a discussion of health care about not really wanting to be rational, I linked him to those two and summarized them like this (goes off and finds the discussion):

I don't really care what you do on [the first thought experiment]. But I care very much what you do on [the second and third]. The importance of logic appears only when you have something that is more important to you than feeling good.

If your goal is to feel good, you can have whatever health system and whatever solution to the trolley problem makes you feel best. I mean, knowing that I didn't let that poor old cancer patient die would make me feel really warm and fuzzy inside too. And I'd also feel really awful about pushing a fat man onto the tracks.

But if your goal is to save lives, you lose the right to do whatever you want, and you'd better start doing what's logical. The logical solution to the two problems does, of course, save more lives than the warm fuzzy alternative.

So the question is: which is more important to you? Feeling good, or saving lives? As Overcoming Bias says:

"You know what? This isn't about your feelings. A human life, with all its joys and all its pains, adding up over the course of decades, is worth far more than your brain's feelings of comfort or discomfort with a plan. Does computing the expected utility feel too cold-blooded for your taste? Well, that feeling isn't even a feather in the scales, when a life is at stake. Just shut up and multiply."

If you're using a different example with something less important than saving lives, maybe switch to something more important in the cosmic scheme of things. I'm very sympathetic to people who say good feelings are more important to them than a few extra bucks, and I don't even think they're being irrational most of the time. The more important the outcome, the more proportionately important rationality becomes than happy feelings.

Comment author: Alicorn 27 October 2009 01:10:35AM *  3 points [-]

My what now? I think that was someone else. I don't think I've been associated with MIT till now.

MIT not only didn't accept me when I applied, they didn't even reject me. I never heard back from them yea or nay at all.

Comment author: Yvain 27 October 2009 01:15:40AM 2 points [-]

That was me.

Of course, irony being what it is, people will now flag the Alicorn - MIT reference as nonfictional, and be referring to Alicorn's MIT example for the rest of LW history :)

Doing your good deed for the day

115 Post author: Yvain 27 October 2009 12:45AM

Interesting new study out on moral behavior. The one sentence summary of the most interesting part is that people who did one good deed were less likely to do another good deed in the near future. They had, quite literally, done their good deed for the day.

In the first part of the study, they showed that people exposed to environmentally friendly, "green" products were more likely to behave nicely. Subjects were asked to rate products in an online store; unbeknownst to them, half were in a condition where the products were environmentally friendly, and the other half in a condition where the products were not. Then they played a Dictator Game. Subjects who had seen environmentally friendly products shared more of their money.

In the second part, instead of just rating the products, they were told to select $25 worth of products to buy from the store. One in twenty five subjects would actually receive the products they'd purchased. Then they, too, played the Dictator Game. Subjects who had bought environmentally friendly products shared less of their money.

In the third part, subjects bought products as before. Then, they participated in a "separate, completely unrelated" experiment "on perception" in which they earned money by identifying dot patterns. The experiment was designed such that participants could lie about their perceptions to earn more. People who purchased the green products were more likely to do so.

This does not prove that environmentalists are actually bad people - remember that whether a subject purchased green products or normal products was completely randomized. It does suggest that people who have done one nice thing feel less of an obligation to do another.

This meshes nicely with a self-signalling conception of morality. If part of the point of behaving morally is to convince yourself that you're a good person, then once you're convinced, behaving morally loses a lot of its value.

By coincidence, a few days after reading this study, I found this article by Dr. Beck, a theologian, complaining about the behavior of churchgoers on Sunday afternoon lunches. He says that in his circles, it's well known that people having lunch after church tend to abuse the waitstaff and tip poorly. And he blames the same mechanism identified by Mazar and Zhong in their Dictator Game. He says that, having proven to their own satisfaction that they are godly and holy people, doing something else godly and holy like being nice to others would be overkill.

It sounds...strangely plausible.

If this is true, then anything that makes people feel moral without actually doing good is no longer a harmless distraction. All those biases that lead people to give time and money and thought to causes that don't really merit them waste not only time and money, but an exhaustible supply of moral fiber (compare to Baumeister's idea of willpower as a limited resource).

People here probably don't have to worry about church. But some of the other activities Dr. Beck mentions as morality sinkholes seem appropriate, with a few of the words changed:

Bible study
Voting Republican
Going on spiritual retreats
Reading religious books
Arguing with evolutionists
Sending your child to a Christian school or providing education at home
Using religious language
Avoiding R-rated movies
Not reading Harry Potter.


Let's not get too carried away with the evils of spiritual behavior - after all, data do show that religious people still give more to non-religious charities than the nonreligious do. But the points in and of themselves are valid. I've seen Michael Keenan and Patri Friedman say exactly the same thing regarding voting, and I would add to the less religion-o-centric list:

Joining "1000000 STRONG AGAINST WORLD HUNGER" type Facebook groups
Reading a book about the struggles faced by poor people, and telling people how emotional it made you
"Raising awareness of problems" without raising awareness of any practical solution
Taking (or teaching) college courses about the struggles of the less fortunate
Many forms of political, religious, and philosophical arguments

My preferred solution to this problem is to consciously try not to count anything I do as charitable or morally relevant except actually donating money to organizations. It is a bit extreme, but, like Eliezer's utilitarian foundation for deontological ethics, sometimes to escape the problems inherent in running on corrupted hardware you have to jettison all the bathwater, even knowing it contains a certain number of babies. A lot probably slips by subconsciously, but I find it better than nothing (at least, I did when I was actually making money; it hasn't worked since I went back to school. Your mileage may vary.

It may be tempting to go from here to a society where we talk much less about morality, especially little bits of morality that have no importance on their own. That might have unintended consequences. Remember that the participants in the study who saw lots of environmentally friendly products but couldn't buy any ended up nicer. The urge to be moral seems to build up by anything priming us with thoughts of morality.

But to prevent that urge from being discharged, we need to plug up the moral sinkholes Dr. Beck mentions, and any other moral sinkholes we can find. We need to give people less moral recognition and acclaim for performing only slightly moral acts. Only then can we concentrate our limited moral fiber on truly improving the world.

And by, "we", I mean "you". I've done my part just by writing this essay.

Solutions to Political Problems As Counterfactuals

37 Post author: Yvain 25 September 2009 05:21PM

A mathematician wakes up to find his house on fire. He frantically looks around before seeing the fire extinguisher on the far wall of the room. "Aha!" he says, "a solution exists!" and goes back to sleep.

    -- Popular math students' joke

There has been much discussion of coulds, woulds, and shoulds recently. Agents imagine different counterfactual states of their own minds or actions, then select the most desirable. Something similar seems to happen during political discussions, but the multiplicity of agents involved muddles it a little.

I recently read a letter to the editor in my local paper. The city was launching a public education campaign against binge drinking, and this letter writer thought that all the billboards and lectures and what-not were a waste of time. She said that instead of a flashy and expensive public awareness campaign, the real solution was for binge drinkers to take responsibility for their own actions and learn that there were ways to have fun that didn't involve alcohol.

This struck me as a misguided line of thinking. Consider this analogy: pretend that the city government was, instead, increasing the number of police to prevent terrorist attacks. And that the writer was arguing that no, we shouldn't get the police involved: the real solution was for terrorists to stop being so violent and attacking people. This would be a weird and completely useless response.

Attempts to solve political problems are counterfactuals in the same way attempts to solve other problems are. In Newcomb's Problem, I modify the "my decision" node and watch what happens to the "money in the box" and "money I get" nodes. When I say "Increasing local police would prevent terrorist attacks," I am modifying the "local police" node, and positing that this would have a certain inhibitory effect on the connected "terrorist attacks" node.

The hypothetical second letter-writer's argument, then, is that if we counterfactually modified the "terrorists' attitude" node, then the "terrorist attacks" node would change. This is correct but useless.

But it's harder to see exactly why it's useless. Consider the original argument "We should raise the number of police and train them in counter-terrorism techniques." In this case, I would be counterfactually modifying the attitudes of (for example) the chief of police. But I'm not the chief of police, any more than I'm Osama bin Laden. If I'm going to let myself modify the chief of police's attitude just because it would be convenient, I might as well let myself turn Osama into a pretty decent guy. Yet "the chief of police should train more policemen" sounds like a potential solution, whereas "terrorists should be nicer" doesn't.

Here's one possible resolution to the problem: it's much more likely I could convince the chief of police to train more policemen than that I could convince terrorists to be nonviolent. Since the police chief shares my goal of stopping terrorists, all I need to do is tell them why training more police would accomplish this goal, and the problem is solved. So the reason why "train more police" counts as a solution is that, as soon as the statement and the evidence supporting the statement reaches the right person, the problem will be solved.

In this case, changing the "chief of police" node is really a proxy for changing the "my actions" node. The solution could be rephrased as "You or I could go to the chief of police and sugget they add more policemen, thus preventing further terrorist attacks." Counterfactually changing your own actions is entirely kosher. This also throws into greater relief the problems with "You or I could find and approach all terrorists and convince them to be nicer," or "You or I could go to every single binge drinker in the city and convince them to be more responsible."1

Actually, when phrased like that, the binge drinking example doesn't sound so bad. Add a comprehensive plan for doing it, enough funding to reach them all, and some idea of how you're going to phrase the "be more responsible" point, and it sounds like, well, a grassroots public awareness campaign. Which is kind of ironic, seeing as the letter started out as an argument against a public awareness campaign, and maybe a sign that I'm taking the Principle of Charity too far here.

...in more realistic situations

It's more complex when there are only small probabilities of your own actions having any effect, but the principle stays the same. For example, I recently heard a doctor say that a single-payer system would best solve the US' health care woes, but since that was politically infeasible he was backing Obama's plan. This one doctor's support will have minimal effect on the chances of Obama's plan passing, but it will have even less of an effect on the chances of single-payer passing. If the expected utilities multiply out in such a way as to make supporting Obama more likely to gain more utility than supporting single-payer, the doctor is justified in his strategy of support for Obama's plan.

One more example from real history I learned recently. Suppose you are a Communist, and your fellow Reds are proposing ways to create a socialist paradise. One says that you must incite the workers to violent revolution. Another says you must petition the current government to support labor reform laws. A third says you must petition the current government to oppose labor reform laws.

Before you expel the third communist from the Party, let them make their argument. They say that the Party doesn't have enough resources to incite violent revolution, and the workers don't want to revolt anyway. Counterfactually modifying the "workers' actions" node to a revolutionary state is a waste of time, because there's no link between any modification of your own actions and that node reaching the state you want. Likewise, modifying "government policy" is useless, because the Communists don't have any clout in the government, so even if you found a wonderful value for that node that would make all workers happy forever, you couldn't change it.

Instead, she says, oppose labor reform laws. These are already unpopular, and even a small party like the Communists would probably have enough power to get them shot down. When there's no labor reform, workers will get angrier and angrier, until they gradually revolt and overthrow the system, getting you what you wanted in the first place.

There were communists in the early 1900s who actually tried this third approach. It didn't work, but I admire their thought processes. They ignored solutions that would never happen, and found an action they thought they could enact, that they thought they would raise the chance of revolution significantly. Compare this kind of cunning to the vapidity of the letter-writer who says "Binge drinkers should become more responsible."

...as an unrealized ideal

I like this way of viewing the problem, because it explains why a certain class of argument feels wrong: arguments that go "The solution to binge drinking is more personal responsibility" or "The solution to poverty is for the poor to work harder," or so on. But do people actually think this way?

The most glaring reason to believe they don't is that most people who "solve" societal problems have no interest in actually enacting their solution. The attitude is something like "Hey, if the federal government passed a single-payer health plan, then all our health troubles would be over!" and then don't bother to write a letter to their representative about it or even convince their next door neighbor.

For reasons that have been discussed ad nauseum on Overcoming Bias, politics is very much a signalling game. In particular, it seems to be a game in which you counterfactually propose different states of the "government policy" node and explain why these would have the best effects, and whoever can give the best explanation gets rewarded with higher status. Sometimes you're also allowed to edit the policies of large private organizations, or of influential individuals. In this case, the problem with the original letter writer wasn't just that she had no plan to enact her solution, but that she was breaking the rule which said that you're only allowed to play with relatively unified, powerful organizations, and not things like "the set of all binge drinkers".

In a way, this isn't so bad. When enough people play this game, their opinions get out to the voters, consumers, politicians, and business leaders, and eventually do change government and private policy.

The point is that if your goal is to actually personally affect things in a direct, immediate way, you can't just apply the rules of this game without thinking beforehand. Communists, when discussing politics for "fun", would never say "I think the government should oppose labor reforms," but that might be the winning move for them when they're actually trying to increase utility. Likewise, libertarians spend a lot of time discussing different ways the government could implement libertarian policy, but when they actually have to take action, the best choice might be seasteads or charter cities or something else that doesn't involve policy at all.

And if you are content to just play the game, at least keep it interesting. No fair counterfactually editing things like "terrorists' behavior" or "poor people's work ethic" or "how responsible binge drinkers are".

Footnotes

1: Or, here's an alternate interpretation of "Binge drinkers should be more responsible": it's not worth trying to prevent binge drinking, because binge drinkers could prevent it themselves if they were more responsible, so it's their own fault. This is not illogical, but applying the argument to a case like "Drunk drivers should be more responsible" would be. There, even if we have no sympathy for drunk drivers, we still need to prevent drunk driving because many of the victims are innocents. The other issue is that people process the two statements "The solution to binge drinking is for binge drinkers to be more responsible" and "We don't need to solve binge drinking; it's the drinkers' own fault and we need not care" differently; the first sounds wise and reasonable, the second callous. For both these reasons, I don't think this interpretation is entirely what the original  letter writer, or other people who use this sort  of argument, are thinking of.

Comment author: Yvain 25 October 2009 10:33:12PM 3 points [-]

I like the thinking behind this post and I think it gives a good reason for reading old books. Upvoted.

But I don't know if you're quite right about nature. This study shows that being in nature provides various hard cognitive benefits. It makes evolutionary sense that we feel more at home in nature and more connected to ourselves there. Why should we be looking for more explanations of why people like nature than that?

I also don't think our views of nature are all that uncontaminated. Yes, there's the animal core of us that thinks "Oh, nature, I understand this." But that's overlaid by a lot of very culturally-determined feelings. In Western culture, the a certain idea of loving wild nature for its own sake really started with the Romantics, and then went through people like Muir to come down to us as various ideas like romanticism, environmentalism, hippie-ism, wilderness sports, and the like. Those counterbalance a whole bunch of other ideas including a medieval Christian/Protestant distrust of wilderness, a Randist "bulldoze it to construct something profitable" ethic, and a whole bunch of other things. I doubt a hippie and an Objectivist would see a waterfall the same way any more than they'd see a strip mall the same way.

I do think everyone including the Objectivist would have a certain biological core set of preprogrammed responses to nature, but I don't know if that's what you're saying.

Comment author: Yvain 22 October 2009 08:53:47PM *  21 points [-]

A great many years ago, a couple of Jehovah Witnesses bit off more than they could chew with my grandmother. During the unsolicited conversation one of them remarked, "Only God can make a rainbow". To which my grandmother-who was watering her plants at the time-said, "Nonsense!", and created her own rainbow with a spray of water from the hose. Family lore has it that was the end of the conversation.

-- seen on Livejournal

Comment author: Gavin 02 October 2009 01:38:05PM 54 points [-]

LW pickup lines:

"Hey baby, wanna get froze together?"

"Wanna make a little adaptation executor? Wanna practice?"

"I don't bite. Unless that's in your definition of Friendliness . . ."

"Wanna grab coffee and swap some memes?"

"There's a world in which we get together. Wanna make it this one?"

"The coin came up heads, so I guess you have to sleep with me."

Comment author: Yvain 04 October 2009 10:35:41AM 70 points [-]

"You must be an expert Bayesian...cause you've got a great posterior."

Comment author: SforSingularity 25 September 2009 07:51:40PM *  34 points [-]

it seems to be a game in which you counterfactually propose different states of the "government policy" node and explain why these would have the best effects, and whoever can give the best explanation gets rewarded with higher status.

no, no. The game is to counterfactually propose different states of the "government policy" node that involve making the government conform more to some ideology X, and then confabulate reasons why this would result in great success. In doing this, you signal your allegiance to ideology X.

But really, the game can work with pretty much anything in the place of the "government policy" node; it can be pretty much any decisionmaking entity, including the companies or diffuse classes of individuals. E.g.

"If binge drinkers went to church more, then they would find the inner strength to overcome the addiction!" (signalling religious allegiance)

"Binge drinkers have the right to run their own lives, the government should keep its hands off them!" (signalling libertarian allegiance)

"Binge drinkers usually come from deprived families and had poor childhoods, it isn't their fault, it's the government's fault for not having enough social welfare!" (signalling liberal allegiance)

"Binge drinking is caused by the breakdown of traditional family values, we need a return to the good-old-fashioned traditional family values!" (Signalling conservative allegiance)

"Binge drinking could be prevented by human neuroenhancements that prevented alcohol from being addictive, we should push for faster research into such technology!" (Signalling h+ allegiance)

Comment author: Yvain 25 September 2009 08:30:42PM 13 points [-]

Oh, very clever. This is a much better explanation for why proposing changes to unchangeable nodes is so well accepted.

Solutions to Political Problems As Counterfactuals

37 Post author: Yvain 25 September 2009 05:21PM

A mathematician wakes up to find his house on fire. He frantically looks around before seeing the fire extinguisher on the far wall of the room. "Aha!" he says, "a solution exists!" and goes back to sleep.

    -- Popular math students' joke

There has been much discussion of coulds, woulds, and shoulds recently. Agents imagine different counterfactual states of their own minds or actions, then select the most desirable. Something similar seems to happen during political discussions, but the multiplicity of agents involved muddles it a little.

I recently read a letter to the editor in my local paper. The city was launching a public education campaign against binge drinking, and this letter writer thought that all the billboards and lectures and what-not were a waste of time. She said that instead of a flashy and expensive public awareness campaign, the real solution was for binge drinkers to take responsibility for their own actions and learn that there were ways to have fun that didn't involve alcohol.

This struck me as a misguided line of thinking. Consider this analogy: pretend that the city government was, instead, increasing the number of police to prevent terrorist attacks. And that the writer was arguing that no, we shouldn't get the police involved: the real solution was for terrorists to stop being so violent and attacking people. This would be a weird and completely useless response.

Attempts to solve political problems are counterfactuals in the same way attempts to solve other problems are. In Newcomb's Problem, I modify the "my decision" node and watch what happens to the "money in the box" and "money I get" nodes. When I say "Increasing local police would prevent terrorist attacks," I am modifying the "local police" node, and positing that this would have a certain inhibitory effect on the connected "terrorist attacks" node.

The hypothetical second letter-writer's argument, then, is that if we counterfactually modified the "terrorists' attitude" node, then the "terrorist attacks" node would change. This is correct but useless.

But it's harder to see exactly why it's useless. Consider the original argument "We should raise the number of police and train them in counter-terrorism techniques." In this case, I would be counterfactually modifying the attitudes of (for example) the chief of police. But I'm not the chief of police, any more than I'm Osama bin Laden. If I'm going to let myself modify the chief of police's attitude just because it would be convenient, I might as well let myself turn Osama into a pretty decent guy. Yet "the chief of police should train more policemen" sounds like a potential solution, whereas "terrorists should be nicer" doesn't.

Here's one possible resolution to the problem: it's much more likely I could convince the chief of police to train more policemen than that I could convince terrorists to be nonviolent. Since the police chief shares my goal of stopping terrorists, all I need to do is tell them why training more police would accomplish this goal, and the problem is solved. So the reason why "train more police" counts as a solution is that, as soon as the statement and the evidence supporting the statement reaches the right person, the problem will be solved.

In this case, changing the "chief of police" node is really a proxy for changing the "my actions" node. The solution could be rephrased as "You or I could go to the chief of police and sugget they add more policemen, thus preventing further terrorist attacks." Counterfactually changing your own actions is entirely kosher. This also throws into greater relief the problems with "You or I could find and approach all terrorists and convince them to be nicer," or "You or I could go to every single binge drinker in the city and convince them to be more responsible."1

Actually, when phrased like that, the binge drinking example doesn't sound so bad. Add a comprehensive plan for doing it, enough funding to reach them all, and some idea of how you're going to phrase the "be more responsible" point, and it sounds like, well, a grassroots public awareness campaign. Which is kind of ironic, seeing as the letter started out as an argument against a public awareness campaign, and maybe a sign that I'm taking the Principle of Charity too far here.

...in more realistic situations

It's more complex when there are only small probabilities of your own actions having any effect, but the principle stays the same. For example, I recently heard a doctor say that a single-payer system would best solve the US' health care woes, but since that was politically infeasible he was backing Obama's plan. This one doctor's support will have minimal effect on the chances of Obama's plan passing, but it will have even less of an effect on the chances of single-payer passing. If the expected utilities multiply out in such a way as to make supporting Obama more likely to gain more utility than supporting single-payer, the doctor is justified in his strategy of support for Obama's plan.

One more example from real history I learned recently. Suppose you are a Communist, and your fellow Reds are proposing ways to create a socialist paradise. One says that you must incite the workers to violent revolution. Another says you must petition the current government to support labor reform laws. A third says you must petition the current government to oppose labor reform laws.

Before you expel the third communist from the Party, let them make their argument. They say that the Party doesn't have enough resources to incite violent revolution, and the workers don't want to revolt anyway. Counterfactually modifying the "workers' actions" node to a revolutionary state is a waste of time, because there's no link between any modification of your own actions and that node reaching the state you want. Likewise, modifying "government policy" is useless, because the Communists don't have any clout in the government, so even if you found a wonderful value for that node that would make all workers happy forever, you couldn't change it.

Instead, she says, oppose labor reform laws. These are already unpopular, and even a small party like the Communists would probably have enough power to get them shot down. When there's no labor reform, workers will get angrier and angrier, until they gradually revolt and overthrow the system, getting you what you wanted in the first place.

There were communists in the early 1900s who actually tried this third approach. It didn't work, but I admire their thought processes. They ignored solutions that would never happen, and found an action they thought they could enact, that they thought they would raise the chance of revolution significantly. Compare this kind of cunning to the vapidity of the letter-writer who says "Binge drinkers should become more responsible."

...as an unrealized ideal

I like this way of viewing the problem, because it explains why a certain class of argument feels wrong: arguments that go "The solution to binge drinking is more personal responsibility" or "The solution to poverty is for the poor to work harder," or so on. But do people actually think this way?

The most glaring reason to believe they don't is that most people who "solve" societal problems have no interest in actually enacting their solution. The attitude is something like "Hey, if the federal government passed a single-payer health plan, then all our health troubles would be over!" and then don't bother to write a letter to their representative about it or even convince their next door neighbor.

For reasons that have been discussed ad nauseum on Overcoming Bias, politics is very much a signalling game. In particular, it seems to be a game in which you counterfactually propose different states of the "government policy" node and explain why these would have the best effects, and whoever can give the best explanation gets rewarded with higher status. Sometimes you're also allowed to edit the policies of large private organizations, or of influential individuals. In this case, the problem with the original letter writer wasn't just that she had no plan to enact her solution, but that she was breaking the rule which said that you're only allowed to play with relatively unified, powerful organizations, and not things like "the set of all binge drinkers".

In a way, this isn't so bad. When enough people play this game, their opinions get out to the voters, consumers, politicians, and business leaders, and eventually do change government and private policy.

The point is that if your goal is to actually personally affect things in a direct, immediate way, you can't just apply the rules of this game without thinking beforehand. Communists, when discussing politics for "fun", would never say "I think the government should oppose labor reforms," but that might be the winning move for them when they're actually trying to increase utility. Likewise, libertarians spend a lot of time discussing different ways the government could implement libertarian policy, but when they actually have to take action, the best choice might be seasteads or charter cities or something else that doesn't involve policy at all.

And if you are content to just play the game, at least keep it interesting. No fair counterfactually editing things like "terrorists' behavior" or "poor people's work ethic" or "how responsible binge drinkers are".

Footnotes

1: Or, here's an alternate interpretation of "Binge drinkers should be more responsible": it's not worth trying to prevent binge drinking, because binge drinkers could prevent it themselves if they were more responsible, so it's their own fault. This is not illogical, but applying the argument to a case like "Drunk drivers should be more responsible" would be. There, even if we have no sympathy for drunk drivers, we still need to prevent drunk driving because many of the victims are innocents. The other issue is that people process the two statements "The solution to binge drinking is for binge drinkers to be more responsible" and "We don't need to solve binge drinking; it's the drinkers' own fault and we need not care" differently; the first sounds wise and reasonable, the second callous. For both these reasons, I don't think this interpretation is entirely what the original  letter writer, or other people who use this sort  of argument, are thinking of.

Comment author: Yvain 24 September 2009 06:05:20PM *  6 points [-]

Do we value saving lives independently of the good feelings we get from it? Some people seem to (or claim to), while others don’t (or claim not to). For those who do, some value (or claim to value) the lives saved linearly, and others don’t.

Perhaps (this is a descriptive and not a normative answer) we value believing we're good people. If we know that we're doing a sub-maximal amount of good in order to feel better, we won't feel like good people anymore. That is, a particular logical sort of person only gets the warm fuzzies when they know they're acting objectively rather than trying to maximize warm fuzzies.

Example: Sally can donate $100/month to charity. She likes dogs, so she donates it to help stray dogs, and feels like a good person. Then someone points out to her that unless she thinks animal lives are more important than human lives, she should donate it to help humans. Now if she donates to animal shelters just because she likes animals, she will feel like a bad person because she knows she's more interested in fuzzy feelings than in actually doing good. Therefore, she donates to human-oriented charities. Now she knows that she's really helping other people instead of just feeling good about herself, and so she feels good about herself. Since she feels good about herself, she keeps doing it.

Comment author: Yvain 23 September 2009 10:08:17PM 16 points [-]

Oh god. The future of the human species is in the hands of a guy who writes crossover fan-fiction.

Comment author: steven0461 23 September 2009 08:00:58PM 8 points [-]

Why can you still use the SIA to prevent Doomsday?

You just did -- early doom and late doom ended up equally probable, where an uncountered Doomsday argument would have said early doom is much more probable (because your living in 2009 is much more probable conditional on early doom than on late doom).

Comment author: Yvain 23 September 2009 08:56:47PM *  3 points [-]

Whoa.

Okay, I'm clearly confused. I was thinking the Doomsday Argument tilted the evidence in one direction, and then the SIA needed to tilt the evidence in the other direction, and worrying about how the SIA doesn't look capable of tilting evidence. I'm not sure why that's the wrong way to look at it, but what you said is definitely right, so I'm making a mistake somewhere. Time to fret over this until it makes sense.

PS: Why are people voting this up?!?

Comment author: Yvain 23 September 2009 07:01:44PM *  7 points [-]

I upvoted this and I think you proved SIA in a very clever way, but I still don't quite understand why SIA counters the Doomsday argument.

Imagine two universes identical to our own up to the present day. One universe is destined to end in 2010 after a hundred billion humans have existed, the other in 3010 after a hundred trillion humans have existed. I agree that knowing nothing, we would expect a random observer to have a thousand times greater chance of living in the long-lasting universe.

But given that we know this particular random observer is alive in 2009, I would think there's an equal chance of them being in both universes, because both universes contain an equal number of people living in 2009. So my knowledge that I'm living in 2009 screens off any information I should be able to get from the SIA about whether the universe ends in 2010 or 3010. Why can you still use the SIA to prevent Doomsday?

[analogy: you have two sets of numbered balls. One is green and numbered from 1 to 10. The other is red and numbered from 1 to 1000. Both sets are mixed together. What's the probability a randomly chosen ball is red? 1000/1010. Now I tell you the ball has number "6" on it. What's the probability it's red? 1/2. In this case, Doomsday argument still applies (any red or green ball will correctly give information about the number of red or green balls) but SIA doesn't (any red or green ball, given that it's a number shared by both red and green, gives no information on whether red or green is larger.)]

Comment author: SforSingularity 18 September 2009 08:37:28PM 1 point [-]

QI doesn't specify. A reasonable assumption would simply be to condition upon your survival, so at 7:59 you assign, say, a 1-10^-6 probability to the coin landing tails, and a 10^-6 probability to other ways you could be saved, for example rescue by an Idiran assault force, a quantum bubble, etc.

Comment author: Yvain 18 September 2009 09:17:03PM 1 point [-]

Well, the reason I ask is that if you're standing outside the multiverse at 8:16, and you count the number of universes with living mes that were saved by the coin landing tails, and those with living mes that were saved by quantum fluctuation, the ones with tails outnumber the ones with fluctuations several gazillion to one, since from an outsider's point of view there's a 1/2 chance I'll get saved by tails versus a one in a gazillion chance I'll be saved by fluctuations.

But from my perspective, if and only if we enforce continuity of experience, there's a 50-50 chance I'll find myself saved by tails vs. fluctuations. But this creates the odd situation of there being certain "mes" in the multiverse whom I am much more likely to end out as than others.

Comment author: cousin_it 18 September 2009 07:50:29PM *  5 points [-]

The water level rises at a rate of 1 foot per minute, so by 8:06 the room is completely flooded with water and you drown and die before 8:15.

No, quantum immortality claims that you won't drown even if the room gets flooded with certainty. You'll be saved by a quantum-fluctuation air bubble or something.

Comment author: Yvain 18 September 2009 08:15:30PM 1 point [-]

So, at 7:59, what probability do you assign to experiencing, at 8:16, a memory of having been saved by the coin landing tails, versus a memory of having been saved by quantum fluctuation?

Comment author: SforSingularity 18 September 2009 07:59:57PM -1 points [-]

you should be able to prove quantum immortality with a quantum random number generator and a bunch of sleeping pills.

so, you hook the pills up to a random number generator: if it outputs 00000000000 you get the placebo, if it outputs anything else, you get sleeping pills. Then check to see if you are awake an hour later. But if you did find yourself still awake, you would remember it in the morning, so the proof would last forever. But since I have already experienced sleep - and lots of it recently - I can conclude that the experiment would almost certainly conclude the boring way.

I suppose "you will experience a string of increasingly unlikley events that seem to be contrived just to keep you alive." would have to include the possibility that you lose consciousness for a long time and then regain it.

Comment author: Yvain 18 September 2009 08:08:15PM 2 points [-]

But if you did find yourself still awake, you would remember it in the morning, so the proof would last forever.

But in the morning, there'd no longer be a link between the existence of your subjective experiences and the unlikely quantum event; you'd be having subjective experiences no matter how the quantum event turned out.

That makes it equivalent to the example where you commit quantum suicide fifty times, survive each time, and then ask another person in the same universe what their belief in quantum immortality is. Since that person saw an unlikely event (you surviving), but it was equally unlikely with or without QI (because your survival is not linked to their current subjective experience) that person can only say "Well, something very unlikely just happened, but it has no bearing on QI."

Same is true of yourself in the morning. Because you'd be having the subjective experience no matter what, all you know in the universe where you stayed up all night was that an unlikely event happened that was equally unlikely with or without QI.

(I'm assuming here that experience doesn't have to be continuous to be experience, and that I am the same person as I was before the last time I went to sleep).

Comment author: Yvain 18 September 2009 07:41:48PM *  6 points [-]

I think I've seen the following argument somewhere, but I can't remember where:

Consider the following villainous arrangement: you are locked in a watertight room, 6 feet high. At 8 o'clock, a computer flips a quantum coin. If it comes up heads, the computer opens a valve, causing water to flow into the room. The water level rises at a rate of 1 foot per minute, so by 8:06 the room is completely flooded with water and you drown and die before 8:15. In either case, the room unlocks automatically by 8:15, so if the coin landed tails you may walk out and continue with your life.

At 7:59, you assign a 50-50 probability that at 8:03, you will experience being up to your waist in water. If you believe in quantum immortality, you will also be assigning a 100 percent chance that at 8:15, you will experience being completely dry and walking out of the room in perfect safety. Since every world-line in which you're in the water at 8:03 is also a world-line in which you're dead at 8:15, these probabilities seem inconsistent.

This suggests that you should consider a 50-50 chance of having your subjective experience snuffed out, since you could find yourself in the water-rising world, and there's nowhere to go from there but death. The other possibility - that you'd be shunted for some reason into a valve-doesn't-open branch even before there's any threat of death - seems to require a little too much advance planning.

But no form of death is instantaneous. Even playing quantum Russian Roulette, there's still a split second between the bullet firing and your death, which is more than enough time to shunt you into a certain world-line.

This doesn't go so well with the adorable zombie-face graphics above. It would go a little better with a model in which, at any point, one copy of you was randomly selected to be the one you're experiencing, but that model has the slight disadvantage of making no sense. it would also go well with a model in which you can just plain die.

Comment author: cousin_it 18 September 2009 06:38:12PM *  4 points [-]

Quantum immortality seems to predict that you can't go to sleep because the conscious phase must go on and on. So it must be wrong. But I don't know where exactly.

Comment author: Yvain 18 September 2009 07:31:45PM 5 points [-]

Or perhaps it only predicts that you will never observe being asleep.

Consider a person who plans to sleep from midnight to eight AM, but will stay up all night if a quantum coin comes up heads. At 11 PM the previous night, the person is awake no matter what, and the coin could be destined to come up either value. At 1 AM, only the people in the universes where the coin came up heads are still having conscious experiences, so they have some evidence that the coin came up heads. At 9 AM, everyone in both universes is having conscious experiences again, and a conscious observer is equally likely to have memories of the coin coming up heads or tails.

Really, all quantum immortality should be able to tell you is that at any moment you are having subjective experience, you are in a universe where events have proceeded in such a way as to keep you awake. So far, it's been startlingly accurate on that account :)

(you should be able to prove quantum immortality with a quantum random number generator and a bunch of sleeping pills. Too bad the proof will only work for a few hours.)

Comment author: Yvain 18 September 2009 01:18:44PM 6 points [-]

However there is a way to convince a lot of people in a few Everett branches: You make a one-time big announcement in the Internet, TV etc. and say that there is a well tested quantum coin-flipper, examined by a community consisting of the most honest and trusted members of the society. You take some random 20 bit number and say that you will flip the equipment 20 times and if the outcome is the same as the predetermined number, then you will take it as a one to million evidence that the Multiple World theory works as expected. Of course, only people in the right branch will be convinced. Nevertheless, they could be convinced enough to make serious thoughts about the viability of quantum Russian roulette type games.

I vaguely remember this being discussed here before, and people deciding it wouldn't work. Before the coin-flipper is run, you have a 1/2^20 chance of seeing your number come up, whether many worlds is true or false. That means that seeing the number come up doesn't tell you anything about whether MW is true or not. It just tells you you're extremely lucky: either lucky enough that the coin-flipper got a very specific number, or lucky enough to have ended up in the very specific universe where the flipper got that number.

Comment author: Yvain 10 September 2009 05:02:50PM *  4 points [-]

More thinking out loud:

It really is in your best interest to accept the offer after you're in a green room. It really is in your best interest to accept the offer conditional on being in a green room before you're assigned. Maybe part of the problem arises because you think your decision will influence the decision of others, ie because you're acting like a timeless decision agent. Replace "me" with "anyone with my platonic computation", and "I should accept the offer conditional on being in a green room" with "anyone with my platonic computation should accept the offer, conditional on anyone with my platonic computation being in a green room." But the chances of someone with my platonic computation being in a green room is 100%. Or, to put it another way, the Platonic Computation is wondering "Should I accept the offer conditional on any one of my instantiations being in a green room?". But the Platonic Computation knows that at least one of its instantiations will be in a green room, so it declines the offer. If the Platonic Computation was really a single organism, its best option would be to single out one of its instantiations before-hand and decide "I will accept the offer, given that Instantiation 6 is in a green room" - but since most instantiations of the computation can't know the status of Instantiation 6 when they decide, it doesn't have this option.

Comment author: Eliezer_Yudkowsky 09 September 2009 08:53:54PM 1 point [-]

Then someone wins the lottery, Omega asks if they're a brain in a vat, and they say yes, and Omega laughs at them

By assumption, if the person is right to believe they're in a sim, then most of the lottery winners are in sims, so while Omega laughs at them in our world, they win the bet with Omega in most of their worlds.

wrong and can't do anything about it

should have been your clue to check further.

Comment author: Yvain 10 September 2009 02:30:44PM *  1 point [-]

This is a feature of the original problem, isn't it?

Let's say there are 1000 brains in vats, each in their own little world, and a "real" world of a billion people. The chance of a vat-brain winning the lottery is 1, and the chance of a real person winning the lottery is 1 in a million. There are 1000 real lottery winners and 1000 vat lottery winners, so if you win the lottery your chance of being in a vat is 50-50. However, if you look at any particular world, the chances of this week's single lottery winner being a brain in a vat is 1000/1001.

Assume the original problem is run multiple times in multiple worlds, and that the value of pi somehow differs in those worlds (probably you used pi precisely so people couldn't do this, but bear with me). Of all the people who wake up in green rooms, 18/20 of them will be right to take your bet. However, in each particular world, the chances of the green room people being right to take the bet is 1/2.

In this situation there is no paradox. Most of the people in the green rooms come out happy that they took the bet. It's only when you limit it to one universe that it becomes a problem. The same is true of the lottery example. When restricted to a single (real, non-vat) universe, it becomes more troublesome.

Comment author: Yvain 09 September 2009 08:07:55PM *  4 points [-]

Curses on this problem; I spent the whole day worrying about it, and am now so much of a wreck that the following may or may not make sense. For better or worse, I came to a similar conclusion of Psy-Kosh: that this could work in less anthropic problems. Here's the equivalent I was using:

Imagine Omega has a coin biased so that it comes up the same way nine out of ten times. You know this, but you don't know which way it's biased. Omega allows you to flip the coin once, and asks for your probability that it's biased in favor of heads. The coin comes up heads. You give your probability as 9/10.

Now Omega takes 20 people and puts them in the same situation as in the original problem. It lets each of them flip their coins. Then it goes to each of the people who got tails, and offers $1 to charity for each coin that came up tails, but threatens to steal $3 from charity for each coin that came up heads.

This nonanthropic problem works the same way as the original anthropic problem. If the coin is really biased heads, 18 people will get heads and 2 people will get tails. In this case,the correct subjective probability to assign is definitely 9/10 in favor of whatever result you got; after all, this is the correct probability when you're the only person in the experiment, and just knowing that 19 other people are also participating in the experiment shouldn't change matters.

I don't have a formal answer for why this happens, but I can think of one more example that might throw a little light on it. In another thread, someone mentioned that lottery winners have excellent evidence that they are brains-in-a-vat and that the rest of the world is an illusion being put on by the Dark Lord of the Matrix for their entertainment. After all, if this was true, it wouldn't be too unlikely for them to win the lottery, so for a sufficiently large lottery, the chance of winning it this way exceeds the chance of winning it through luck.

Suppose Bob has won the lottery and so believes himself to be a brain in a vat. And suppose that the evidence for the simulation argument is poor enough that there is no other good reason to believe yourself to be a brain in a vat. Omega goes up to Bob and asks him to take a bet on whether he is a brain in a vat. Bob says he is, he loses, and Omega laughs at him. What did he do wrong? Nothing. Omega was just being mean by specifically asking the one person whom ve knew would get the answer wrong.

Omega's little prank would still work if ve announced ver intention to perform it beforehand. Ve would say "When one of you wins the lottery, I will be asking this person to take a bet whether they are a brain in a vat or not!" Everyone would say "That lottery winner shouldn't accept Omega's bet. We know we're not brains in vats." Then someone wins the lottery, Omega asks if they're a brain in a vat, and they say yes, and Omega laughs at them (note that this also works if we consider a coin with a bias such that it lands the same way 999999 out of a million times, let a million people flip it once, and ask people what they think the coin's bias is, asking the people who get the counter-to-expectations result more often than chance.)

Omega's being equally mean in the original problem. There's a 50% chance ve will go and ask the two out of twenty people who are specifically most likely to be wrong and can't do anything about it. The best course I can think of would be for everyone to swear an oath not to take the offer before they got assigned into rooms.

Comment author: Yvain 08 September 2009 09:03:15AM 6 points [-]

Thanks for this sequence. I wasn't really able to follow some of the other decision theory posts, but this is very clear, ties a lot of things together, and finally gives me a good handle on what a lot of people are talking about.

Comment author: taw 06 September 2009 05:06:55PM 0 points [-]

Is overhead of measuring time spent on a particular activity a big problem, especially if you're multitasking like most people?

Comment author: Yvain 07 September 2009 09:09:55AM 0 points [-]

Time budgeting hasn't been a problem; it's pretty easy to keep track of.

Comment author: Yvain 06 September 2009 01:49:41PM *  1 point [-]

I'm currently doing something similar to this as the natural progression of the method I described here. The main difference is that instead of maximizing points, I'm holding myself to a budget of points.

It started by me telling myself I was going to spend two hundred fifty minutes / day doing useful self-improvement type activities. Then I decided that certain activities were only marginally useful, and started counting them at a rate of one minute per two minutes, and others were extremely useful, and counting them at a rate of two minutes per minute, and now it's more or less the same as your point system. I also subtract points for certain things I want to do less of. I've been doing it for about a month now pretty successfully.

At some point I want to make a post on it, but not until I've got more to say about it. I'm trying to think of it with an economic metaphor, as the personal equivalent of subsidizing useful activities and taxing useless activities to change the resources devoted to each, but I can't really present the analogy coherently until I understand the mind better.

Comment author: Yvain 05 September 2009 12:55:43PM *  6 points [-]

In my experience, beginning math students simply expect their nice-sounding procedures to work. For example, they expect to be able to add fractions straight across. When you tell them they can’t, they demand to know why they can’t, as though most nice-sounding theorems are true, and if you want to claim that one isn’t, the burden of proof is on you. It is only after students gain considerable mathematical sophistication (or experience getting burned by expectations that don’t pan out) that they place the burden of proofs on the theorems.

-- Anna Salamon

Comment author: Yvain 03 September 2009 03:09:19PM *  6 points [-]

"What's that saying?" he said, smiling crookedly. "When you've eliminated the impossible, whatever it is that remains--- "

"--- however improbable, must be the truth. Yes, the problem is, the man who wrote that believed in faeries, and that he could photograph them."

  • S. M. Stirling, The Peshawar Lancers
Comment author: Yvain 29 August 2009 12:41:31PM 4 points [-]

If I wanted to test sociobiology as a theory with these statistics, I'd be much more interested in the numbers for hunter-gatherer tribes. In particular, I expect the invention of birth control has driven these numbers well away from what sociobiology would predict.

I guess these are a test of whether sociobiology can be applied willy-nilly to make predictions about every facet of our own society, but I hope people don't think it's that easy.

Comment author: timtyler 21 August 2009 08:16:01PM -1 points [-]

I feel like I already addressed such issues when I wrote: "We do not have to choose between these two theories." Sometimes the conscious goals are best, and sometimes the unconscious ones are. You have given some examples of the former, but there are also examples of the latter.

Comment author: Yvain 21 August 2009 09:54:39PM 0 points [-]

Sorry, long time ago and different section of the comments. I think with that clarified I mostly agree with you anyway.

Comment author: Simon_Jester 21 August 2009 04:56:49PM *  1 point [-]

One thing that caught my eye is the presentation of "Universe is not filled with technical civilizations..." as data against the hypothesis of modern civilizations being probable.

It occurs to me that this could mean any of three things, which only one of which indicates that modern civilizations are improbable.

1) Modern civilizations are in fact as rare as they appear to be because they are unlikely to emerge. This is the interpretation used by this article.

2) Modern civilizations collapse quickly back to a premodern state, either by fighting a very destructive war, by high-probability natural disasters, by running out of critical resources, or by a cataclysmic industrial accident such as major climate change or a Gray Goo event.

This would undermine an attempt to judge the odds of modern civilizations emerging based on a small sample size. If (2) is true, the fact that we haven't seen a modern civilization doesn't mean it doesn't exist; it's more likely to mean that it didn't last long enough to appear on our metaphorical radar. All we know with high confidence is that there haven't been any modern civilizations on Earth before us, which places an upper bound on the likely range of probabilities for it to happen; Earth may be a late bloomer, but it's unlikely to be such a late bloomer that three or four civilizations would have had time to emerge before we got here.

3) The apparent rarity of modern civilizations could just be a sign that we are bad at detecting them. We know that alien civilizations haven't visited us in the historic past, that they haven't colonized Earth before we got here, and that they haven't beamed detectable transmissions at us, but those quite plausibly be explained by other factors. Some hypotheses come to mind for me, but I removed them for the sake of brevity; they are available if anyone's interested.


Anyway, where I was going with all this: I can see a lot of alternate interpretations to explain the fact that we haven't detected evidence of modern civilizations in our galaxy, some of which would make it hard to infer anything about the likelihood of civilizations emerging from the history of our own planet. That doesn't mean I think that considering the problem isn't worthwhile, though.

Comment author: Yvain 21 August 2009 09:01:38PM *  8 points [-]

4) There is a very easy and unavoidable way to destroy the universe (or make it inhospitable) using technology, and any technological civilization will inevitably do so at a certain pretty early point in its history. Therefore, only one technological civilization per universe ever exists, and we should not be surprised to find ourselves to be the first.

5) The Dark Lords of the Matrix are only interested in running one civilization in our particular sim.

Comment author: pjeby 19 August 2009 02:19:40PM 3 points [-]

The bit of the mind that switches attention around might itself be relatively small - and gains the illusion of size by being able to illuminate many areas of the mind - by damping down perceptions from everywhere else.

And the PCT hypothesis for why this is so (predating the Society of Mind by a decade or so), is that consciousness is effectively the debugger or test rig for the rest of the brain: a tool whose job is the tuning, adjustment, and extension of the brain's unconscious control systems. The conscious mind is heavily engaged in any sort of skill acquisition, "noticing" what perceptions are associated with success or failure, and this noticing process is key to wiring up new control circuits.

From this perspective, consciousness is effectively an on-call maintenance person, a tech support rep for the unconscious. Which provides a good evolutionary reason for "higher" animals to have higher degrees of consciousness; the more flexible the creature, the more advanced the tech support required. ;-)

That humans have decided to rebel and take over the company instead of functioning in a strictly support capacity is a separate issue.

And when the revolution isn't going so well, we call it "akrasia".

So the key to a smooth takeover is realizing that if the unconscious machinery isn't working well, then you will suffer right along with your unconscious. You need a win-win solution, and the unconscious is pretty easily satisfied, being just a big dumb array of thermostats and all.

An array which -- being that you're its tech support rep -- you can actually rewire. In fact, most of what's in there, you consciously put there at some point, or at least didn't object to.

But if you treat it like it's an independent mind -- which it isn't -- and an enemy (which it also isn't) whose demands should be disregarded, then you're never even going to perceive what is actually going on in there, and therefore won't be able to tell how to change any of it. And you'll just keep fighting, instead of debugging.

Not really a good use of your time, IMO.

Comment author: Yvain 21 August 2009 05:43:35PM *  5 points [-]

I think we agree. Your statement that the unconscious is "just a big dumb array of thermostats" is just what I was trying to get across, plus as you said that it isn't an independent mind.

I interpreted Robin (I'm still not sure if I'm right) as suggesting the unconscious is a full and separate mind whose preferences deserve respect for the same reason you'd respect another human's preferences. So that, for example, if you wanted to stay sober but your unconscious wanted to drink, you "owe" it to your unconscious to compromise, in the same way you'd be a bad friend if you didn't take a friend's preferences into account. All I am trying to say is that the unconscious doesn't deserve that kind of respect.

If you're saying that my conscious mind can achieve its own goals better by working with the unconscious in some particular way, well, you're the expert on that and I believe you.

Comment author: timtyler 18 August 2009 07:30:06PM 0 points [-]

Re: I question whether the unconscious is more than a collection of very sophisticated mental modules, in the same way that a bird's brain may have a flight dynamics module, an astronomical navigation module, a mate-preference-analysis module, and so on.

...and what do you think your conscious mind is, then - if not a collection of sophisticated mental modules?

Comment author: Yvain 18 August 2009 09:51:58PM *  0 points [-]

Wikipedia gives Fodor's list of eight characteristics of "mental modules", which include "domain specificity", "fast speed", "shallow output", "limited accessibility", "encapsulation", et cetera, and quotes someone else as saying the most important distinguishing feature is "cognitive impenetrability".

In other words, "module" has a special definition that doesn't mean exactly the same as "something in the mind". So when I "accuse" the unconscious of being "modules", all I'm saying is that it's a bunch of single-purpose unlinked programs, as opposed to the generic and unified programs that make up the conscious mind. This seems relevant since it makes it harder to accept the idea of the unconscious as a separate but equal person living inside your brain.

If there are other definitions of "module" that include anything in the mind, and you're using one of those, then yes, the conscious mind is a module or collection of modules as well.

In response to Experiential Pica
Comment author: BrandonReinhart 18 August 2009 07:16:31AM 12 points [-]

We've gone out on a limb with the akrasia discussion. Posts seem like they are using loose references to published material to justify grand pet theories. The term akrasia has become a tent under which all manner of effects and phenomena are being housed. A counter-reductionist trend has set in. The definition of akrasia grows, the manners of dealing with it expand, possible theories of methods of its identification and mitigation can be found in every other top level post. It has the feel of rat hole.

There is too much explanation and not enough prediction. What am I to anticipate?

Solving the problem of the erosion of will and of short term preferences is going to take more than pouring cold water into our own ears. It is likely not generally solvable through introspection.

This discussion of akrasia afflicts this community like a kind of akrasia -- a fertile field for arm chair theories to distract us from harder problems.

Meditation? Really? "Reset my experiential pica?" What?

I propose a one month ban on the akrasia topic. If there is something to be gained through introspection on the subject let those with an interest introspect. For now, though, the only thing these posts lead me to anticipate is a post titled "Akrasia: Because of Magic!"

Comment author: Yvain 18 August 2009 07:59:35PM 13 points [-]

I was thinking of posting a reply to this like "Hey, this is all very interesting, but you have no evidence whatsoever for it." Decided against it, because it's not being presented as a proven theory. It's being presented as an interesting and elegant possibility that deserves further discussion, or a fertile direction for future research. That puts it on the same footing as eg superstrings, and sometimes one of those sorts of things is the creative seed someone else needs.

No one's going to discover a simple cure-all that works for everyone, but I've already [gotten something really useful to me] partly out of our discussions here, and the comments there make it look like some other people have done the same.

Even if we don't expect any further direct benefits, it might end up kind of like the moonshot, which wasn't too useful in itself but which more than made back its cost in generating peripheral technology. The akrasia discussion has led to some really good peripheral posts like Utilons vs. Hedons, which clarified a lot for me, and some very introductory discussion of PCT, which pjeby thinks explains everything about everything. The comments in my Preferences thread are making me think harder about the differences between the conscious and unconscious than I've ever done before.

And it's not like this is costing us anything. There aren't many posts on LW as it is, you can ignore any you don't like, and the limiting resource on working more on "harder" questions is less time and attention than it is intelligence and inspiration.

Comment author: Yvain 17 August 2009 05:40:22PM *  4 points [-]

The summary of happiness research is good, and the links are good. But one or two parts, mostly the 1st paragraph, seem cliched and preachy.

How, exactly, do I avoid this pitfall of seeking happiness? Should I deliberately take a boring job instead of a fun one? Spend money on things I don't want rather than things I do? The end of the article seems to rephrase this as "seek happiness from eudaimonic pursuits instead of hedonic pursuits, because you get more of it that way", which I would be more on board with.

One thing that bothers me about some happiness research is the very muddled definition of happiness; it seems to switch from "pure pleasure" through various emotions all the way to "utility" at times. In fact, the whole happiness-utility relationship is one I've never seen fully clarified (though this thread makes a really good start). Talking about how much better it is to "live life in accordance with your desires and values" could mean that people should care about utility more than happiness, that utility is equivalent to some kind of happiness, or that increasing utility automatically leads to increasing happiness.

Comment author: SilasBarta 12 August 2009 04:30:06AM 3 points [-]

irrationality of unconscious mind continuing to pursue subgoals when clearly no longer connected to supergoals, unconscious' vulnerability to proximity/scale biases when dealing with morality, and several others.

The conscious is guilty of these too.

I've read plenty about the unconscious, and I admit it's astonishingly complex and capable. So are honeybees. But when bees perform unbelievably complicated tasks, I don't assume they therefore have human-level intelligence, and I think the unconscious' actions are more like the honeybees' than people's.

Okay, but carrying the analogy over, I'm sure you also don't trivialize the value of honey!

However, if there's something you think I should know more about, why not recommend me specific articles, authors, or books?

You could start with making yourself aware of the non-conscious mind's ability to solve CAPTCHAs, an AI-complete problem, and current conscious minds' inability to figure out how they do it with enough clarity to re-implement it in software.

Comment author: Yvain 17 August 2009 04:57:59PM *  4 points [-]

Actually, it's funny you mention CAPTCHAs as your example. If you're going to go that far, why not also attribute skill at chess to the unconscious? After all, it's got to be the unconscious that screens out most of the several dozen possible chess moves each turn and lets your conscious concentrate on the few best, and you can generalize from chess to practically all matters of strategy. Or for that matter, how about language? All my knowledge of English grammar was purely unconscious until I started studying the subject in high school, and 99% of my grammar use still comes from there.

So the issue's not whether it can perform complex tasks. I don't know exactly what the issue is, but I think it connects to the concept of "personhood" somehow. I question whether the unconscious is more than a collection of very sophisticated mental modules, in the same way that a bird's brain may have a flight dynamics module, an astronomical navigation module, a mate-preference-analysis module, and so on.

The computing hardware of my brain contains a program for recognizing letters, a program that detects potential mates and responds with feelings of lust, a program that interacts with my reward system in such a way as to potentially create alcoholism, and so on. They're all computationally very impressive. But I don't see why I should assign them moral status any more than I would feel morally obligated to listen to a laptop on which I had installed a program that detected the presence of beautiful women nearby and then displayed the words "mate with this woman". I don't want to privilege these programs just because they happen to be located inside a human brain and they get reflected glory from some of the other things human brains can do.

To make me want to assign them moral status, you'd have to give me evidence that there was something that it felt like to be my lust. This seems kind of category-error-ish to me. I feel my lust, but my lust itself doesn't feel anything. You may feel sorry for me for having to deal with my lust, but feeling sorry for my lust because I don't choose to satisfy it is in my opinion a waste of sorrow. It's also an infinite regress. If I feel unhappy because I have unfulfilled desire, and my desire feels unhappy because it's unfulfilled, does my desire's unhappiness feel something? Why stop there?

I have a feeling this problem requires more rigor than I can throw at it right now. I've been trying to think about it more clearly so as to hopefully eventually get some top-level posts out of it, but this is the best I can do at the moment.

Comment author: RobinHanson 17 August 2009 01:14:21PM 1 point [-]

Just because your conscious mind isn't aware of experiences by your unconscious mind doesn't mean they don't exit. And just because your unconscious is subject to some biases doesn't mean your conscious mind does better on average.

Comment author: Yvain 17 August 2009 04:40:33PM 2 points [-]

I don't disagree with any of that, but it's all phrased sort of as "you can't prove it doesn't." What should make me single out the hypothesis that it does as worthy of further consideration?

Comment author: RobinHanson 12 August 2009 02:35:55AM 1 point [-]

I didn't see you offering reasons - I just saw you declaring that in general the conscious is more rational and moral, this conclusion being so obvious it didn't need reasons. You later gave specific examples of beliefs in yourself where your conscious part thinks your conscious beliefs are more correct and more moral than your unconscious beliefs, but surely you can't expect that to be considered a sufficient argument about the general trend in all people on all topics.

I do think you could stand to read a bit more about the unconscious; I think you will find it far more complex and capable than you realize.

Comment author: Yvain 12 August 2009 04:23:30AM 8 points [-]

Things I gave as evidence: the logical inconsistency of unconscious mind having conscious experience, irrationality of unconscious mind continuing to pursue subgoals when clearly no longer connected to supergoals, unconscious' vulnerability to proximity/scale biases when dealing with morality, and several others. I don't see how any of these can be dismissed as just "my conscious part thinks my conscious beliefs are more correct" with anything other than a Fully General Counterargument.

I've read plenty about the unconscious, and I admit it's astonishingly complex and capable. So are honeybees. But when bees perform unbelievably complicated tasks, I don't assume they therefore have human-level intelligence, and I think the unconscious' actions are more like the honeybees' than people's.

However, if there's something you think I should know more about, why not recommend me specific articles, authors, or books?

Comment author: Alicorn 10 August 2009 11:10:40PM 4 points [-]

Has anybody ever tried installing a little camera above a stove with a live feed (checkable by cell phone) to see if that helped people panicking about having left it on?

Comment author: Yvain 11 August 2009 12:30:36AM 15 points [-]

I sometimes control my OCD by crossing my fingers in a certain very odd pattern whenever I do something (like locking a door). Then, as soon as I come to a notebook or a computer with my fingers still crossed, I write down "I locked the door at 5:27 PM, August 10, 2009". When I've done this, I can just look at the paper and my compulsion to check whether or not the door is really locked mostly goes away.

I used to try the same thing without the finger crossing, and I found that I was always able to believe there was just the tiniest chance that I might have formed the false memory of having locked the door between locking it and reaching the notepad. Because I don't cross my fingers except while in the act of locking the door and I don't write a note unless my fingers are crossed, I can dispel that last nagging doubt. From a rational point of view it's not very sensible, but it seems to work okay.

Comment author: RobinHanson 10 August 2009 07:52:35PM 1 point [-]

So to summarize, you think your conscious mind is more rational, moral, and qualia-full than your unconscious mind, and the evidence you cite for these conclusions is: your conscious mind has these opinions. Have I got this right? Any idea what opinions your unconscious mind has on these matters?

Comment author: Yvain 10 August 2009 08:17:59PM *  7 points [-]

You can phrase any argument that way:

"You think you're more likely to be correct about evolution than a creationist, and the evidence you cite for this conclusion is that this is your opinion."

Yes, it's my opinion. But it's my opinion for reasons. I thought I gave some good reasons why I thought my conscious mind makes better decisions than my unconscious. You rejected those because it was my conscious mind giving them. But if that were sufficient criteria to reject reasons, you would have to reject everyone's reasons on any subject, from evolution to the roundness of the Earth.

Even aside from all those reasons, I haven't heard any reasons to think the unconscious is aware, rational, or morally reflective, and I think the burden of proof is on that position, since it would basically be saying there's a second person inside my head.

As for what opinion my unconscious mind has, yes, I have some idea. I predict it has no opinions at all, and is more of a collection of drives, instincts, and processes than the sort of entity that forms rational opinions on complicated issues. I doubt my unconscious "disagrees" with me about this any more than my kidneys disagree with my position on tax reform, or my tibia disagrees with my interpretation of quantum mechanics.

If I had multiple personality/dissociative identity disorder, I would be prepared to treat my alternate personalities as worthy of respect and cooperation. But I think my unconscious is probably more like a kidney or a tibia than like a whole other personality. I realize this is a factual claim, and am willing to change my mind if I hear evidence that suggests otherwise.

Comment author: RobinHanson 10 August 2009 03:22:19PM 2 points [-]

The words "conscious" and "unconscious" are widely used; I don't think it helps for you to make up your own definitions. Your evidence about the rationality and morality of your conscious mind come many from your personal conscious beliefs about those features within yourself; these could easily be biased and self-serving. I'm still not entirely clear on what qualia mean, but from what I do understand about them I don't see why the parts of your mind other than the part I'm talking to couldn't have them.

Comment author: Yvain 10 August 2009 07:14:47PM *  5 points [-]

I thought I was listing the standard definition of "conscious" and "unconscious" in a way that made it clearer why they led to my conclusion. If you have a different definition, what is it?

My beliefs about the rationality and morality of my unconscious mind do come from my conscious mind, this is where all my beliefs come from. When I say that a creationist is less rational than I am, or a Nazi is less moral than I am, I'm using those same beliefs from my conscious mind, and they are subject to the same biases. I have to either forfeit my right to judge entirely, or use those same judgments to judge my unconscious. I've learned (partly from you) ways to try to be less biased, and I try to use them here, but in the end all I have is reflective coherence

Just to clarify your position, are you suggesting I have a moral duty to respect my unconscious mind's preferences, in the same way I would have to respect the preferences of another person? Or are you suggesting it would benefit my conscious mind to have inner peace?

Comment author: pjeby 10 August 2009 03:16:42AM 3 points [-]

The only difference is that he isn't suggesting we jettison the ideas of "conscious" and "unconscious", and you are. Generally, if you decide to declare that a principle embraced by several generations of scientists is rot, you should provide some evidence.

Generally, if you decide to pick a point of debate like that, you should try doing some reading first. May I suggest http://en.wikipedia.org/wiki/Unconscious_mind for starters? In particular, the sections on "Controversy" and the notes that mention the current parlance for other-than-conscious mental processes is usually "non-conscious" rather than "unconscious"... precisely because cognitive scientists have found the now-cultural notion of an "unconscious mind" to be misleading about the nature of our non-conscious processes.

And rationalists in particular should be wary of the phrase, because it basically amounts to a stop-sign for actually thinking or investigating anything. I'm reminded of an exchange between a student and Richard Bandler, where the student (a psychiatrist, I believe) asked if something that the demonstration subject was doing was "turning it over to an unconscious process", and Bandler replied with something to the effect of, "That's not actually an explanation, you know. Everything is unconscious until you pay attention to it."

Comment author: Yvain 10 August 2009 04:43:22AM *  6 points [-]

But there are some things on which the attention can easily be focused at will (like the name of your second grade teacher, when you're not thinking about them) and other things upon which the attention can never be focused, or only with great training (like the regulation of body temperature).

And there are some things which it seems like you can change at will (like whether or not you go out to dinner tonight) and other things which it seems you cannot change without great difficulty (like whether you freeze up and "choke" when speaking to large groups of people)?

Aren't priming, response to the IAT, self-handicapping and a slew of other mental phenomena done on a level that cannot be accessed no matter how hard you try to access it?

So what's wrong with going ahead and calling all these things you're not conscious of and cannot choose to focus attention on "unconscious"?

Comment author: RobinHanson 09 August 2009 02:30:54AM *  12 points [-]

You misremember my position. I have not argued that the unconscious is right and the conscious wrong. I have argued (e.g., here) for trying to find a compromise to make peace between the conflicting parts of yourself. At the last OB/LW meetup I argued this point in person to several people, all of whom instead favored vigilant internal war.

Comment author: Yvain 10 August 2009 04:32:26AM 0 points [-]

Sorry. Corrected.

Comment author: pjeby 09 August 2009 02:36:22AM 27 points [-]

In both cases, the conscious mind determines the signal and the unconscious mind determines the action.

Look more closely. All preferences are equal, in the sense of being within the same system -- and this includes signaling preferences. The drunk prefers to drink and prefers to not be thought of as preferring that. But these are not concepts of a different nature; they can both be expressed within the same behavioral preference system.

IOW, both the Cynical and Naive theories are wrong; we only have one set of preferences, it just sometimes works out that the "best" compromise (in the sense of being an approach that your brain can discover through trial and error) is to say one thing and do another. But both the saying and doing are behaviors of the same type; "conscious" vs. "unconscious" is a red herring here.

Now, if you want to say that you don't consciously identify with some subset of your choices or preferences, that's fine, but it's not useful to claim that this is the result of some schism in your being. It's all you, you just aren't being conscious of that part of "you" at the moment.

The "unconscious mind" isn't a real entity; it's not a "mind", in the anthropomorphic sense. It's just whatever you're not paying attention to right now, that keeps on going. If you pay attention to your breathing or your heart rate you can learn to control them. If you pay attention to your feet you'll know where they are right now. And if you pay attention to what you actually get from your "akrasic" behavior, you'll realize it's something you genuinely want.

What you haven't been doing, is negotiating fairly among all your "interests" (to use Ainslie's jargon), or cleanly prioritizing your "controlled variables" (to use Powers's).

Comment author: Yvain 10 August 2009 03:51:05AM 4 points [-]

The "unconscious mind" isn't a real entity; it's not a "mind", in the anthropomorphic sense. It's just whatever you're not paying attention to right now, that keeps on going. If you pay attention to your breathing or your heart rate you can learn to control them. If you pay attention to your feet you'll know where they are right now. And if you pay attention to what you actually get from your "akrasic" behavior, you'll realize it's something you genuinely want.

Patri Friedman once wrote something about "wanting" versus "wanting to want". I agree that everything you do, you genuinely want to do, in the sense that you're not doing it under duress. But not everything you do is something you want to want to do.

Likewise, if I imagine myself suddenly getting infinite willpower, there are certain things I would do more and other things I would do less.

I'm using the word "conscious" to refer to things I want to want and things I would do more with infinite willpower. I'm using the word "unconscious" to refer to things I don't want to want and things I would do less with infinite willpower. I don't think it's too controversial that those are two different categories.

Comment author: RobinHanson 09 August 2009 02:26:46AM *  10 points [-]

Conscious minds are potentially rational, informed by morality, and qualia-laden. Unconscious minds aren't.

Your entire argument for preferring conscious over unconscious minds is this last quick throw away sentence? That's it? Come on, why can't unconscious minds be rational, informed by morality, or qualia-laden? And why are those the features that matter? Are you really implying this is so completely obvious that this one quick sentence is all that needs to be said? Declaring conscious goals to be the goals of the "person", versus unconscious goals as goals of the genome, just presupposes your answer.

Comment author: Yvain 10 August 2009 03:29:09AM 16 points [-]

I guess I did consider it that completely obvious. If it's causing so much controversy, maybe I need to think about it more.

I'm defining my "conscious self" as the part of my mind that creates my verbal stream of thought and which controls what I believe I would do if I had infinite willpower. I'm defining "unconscious self" as the source of my inability to always go through with my conscious mind's desires.

By definition, my unconscious mind has no qualia / experiences / awareness, because if it did it would be part of my conscious mind (I suppose it's possible that it is a "different person" who has experiences that are not my experiences, but I have never heard anyone propose this before and don't know of any evidence for it.)

When I use the word "I", I refer to the locus of my qualia and experiences, and thus to my conscious mind. I have no selfish reason to care about my unconscious mind, because its state as happy or unhappy has no relationship to my state as happy or unhappy except insofar as the unconscious mind can influence the conscious mind. And I have no moral reason to care about my unconscious mind, because in my moral system only aware beings deserve moral consideration; the unconscious mind has no more awareness than a rock and deserves no more moral consideration than a rock does.

Along with my qualia, I identify with my rationality. My rationality is what tells me that there's very probably no such thing as ghosts. This satisfies my conscious mind, which then accepts that there's no such thing as ghosts. It does not satisfy my unconscious mind, which continues to make me flee haunted mansions or sleep with the lights on or something. My rationality is what tells me that I should ask that girl out because the worst she could do is say no. My conscious mind accepts that. My unconscious mind continues to use all of its resources to hold me back from asking.

It seems vanishingly unlikely that my unconscious actually has as supergoals "Flee haunted mansions" and "Never ask girls out" and is rationally achieving them. It seems much more likely that the unconscious is enacting genetic directives like "Avoid danger" and "Avoid taking risks that could lower your social status", but is too irrational to realize that although equivalents of these situations might have been problems in the EEA, they are no longer problems today. It thinks that "Flee haunted mansions" and "Never ask girls out" are appropriate subgoals of the supergoals "Avoid danger" and "Avoid taking risks that could lower your social status", but in fact they aren't. Since it's too dumb to realize this, I feel suitably superior to it to ignore its opinions.

The same is true of morality. My unconscious is what tells me to value the life of a photogenic American more than the life of a starving Ethiopian, to value the life of one specific person more than the life of fifty statistical people, to refuse to push the fat man onto the tracks in the trolley problem no matter how many lives it would save, et cetera. If another person had this morality, I wouldn't respect it in them, and if my own unconscious has this morality, I don't respect it in it either.

Let me also admit that I have a bias here. I've got obsessive-compulsive disorder. It means that my unconscious mind frequently tells me things like "Close that door there eighty two times, or I will throw a fit and not let you feel comfortable for the rest of the day." I know that feeling is caused by miswired circuits in the basal ganglia. Why should I give miswired circuits in the basal ganglia the same respect as I give myself, a full intelligent human being?

All of my other unconscious urges seem closer to that urge to close the door eighty-two times than they do to anything rational or worth respecting.

Comment author: Yvain 08 August 2009 11:21:40PM 9 points [-]

I've heard it said somewhere (sorry, can't come up with a citation) that this is the rationale behind the brain's tendency to actively erase dreams, the bane of dream-diary keepers everywhere. Because we're somewhat source-blind, we risk keeping our attachment to conclusions we made in dreams even after we realize we were dreaming. To prevent that, the brain erases most of the five or so dreams we have each night, and even the ones we remember on waking tend to be mostly erased in a few hours unless you consciously think about them very hard.

Would Your Real Preferences Please Stand Up?

42 Post author: Yvain 08 August 2009 10:57PM

Related to: Cynicism in Ev Psych and Econ

In Finding the Source, a commenter says:

I have begun wondering whether claiming to be victim of 'akrasia' might just be a way of admitting that your real preferences, as revealed in your actions, don't match the preferences you want to signal (believing what you want to signal, even if untrue, makes the signals more effective).

I think I've seen Robin put forth something like this argument [EDIT: Something related, but very different], and TGGP points out that Brian Caplan explicitly believes pretty much the same thing1:

I've previously argued that much - perhaps most - talk about "self-control" problems reflects social desirability bias rather than genuine inner conflict.

Part of the reason why people who spend a lot of time and money on socially disapproved behaviors say they "want to change" is that that's what they're supposed to say.

Think of it this way: A guy loses his wife and kids because he's a drunk. Suppose he sincerely prefers alcohol to his wife and kids. He still probably won't admit it, because people judge a sinner even more harshly if he is unrepentent. The drunk who says "I was such a fool!" gets some pity; the drunk who says "I like Jack Daniels better than my wife and kids" gets horrified looks. And either way, he can keep drinking.

I'll call this the Cynic's Theory of Akrasia, as opposed to the Naive Theory. I used to think it was plausible. Now that I think about it a little more, I find it meaningless. Here's what changed my mind.

What part of the mind, exactly, prefers a socially unacceptable activity (like drinking whiskey or browsing Reddit) to an acceptable activity (like having a wife and kids, or studying)? The conscious mind? As Bill said in his comment, it doesn't seem like it works this way. I've had akrasia myself, and I never consciously think "Wow, I really like browsing Reddit...but I'll trick everyone else into thinking I'd rather be studying so I get more respect. Ha ha! The fools will never see it coming!"

No, my conscious mind fully believes that I would rather be studying2. And this even gets reflected in my actions. I've tried anti-procrastination techniques, both successfully and unsuccessfully, without ever telling them to another living soul. People trying to diet don't take out the cupcakes as soon as no one else is looking (or, if they do, they feel guilty about it).

This is as it should be. It is a classic finding in evolutionary psychology: the person who wants to fool others begins by fooling themselves. Some people even call the conscious mind the "public relations officer" of the brain, and argue that its entire point is to sit around and get fooled by everything we want to signal. As Bill said, "believing the signals, even if untrue, makes the signals more effective."

Now we have enough information to see why the Cynic's Theory is equivalent to the Naive Theory.

The Naive Theory says that you really want to stop drinking, but some force from your unconscious mind is hijacking your actions. The Cynic's Theory says that you really want to keep drinking, but your conscious mind is hijacking your thoughts and making you think otherwise.

In both cases, the conscious mind determines the signal and the unconscious mind determines the action. The only difference is which preference we define as "real" and worthy of sympathy. In the Naive Theory, we sympathize with the conscious mind, and the problem is the unconscious mind keeps committing contradictory actions. In the Cynic's Theory, we symapthize with the unconscious mind, and the problem is the conscious mind keeps sending out contradictory signals. The Naive say: find some way to make the unconscious mind stop hijacking actions! The Cynic says: find some way to make the conscious mind stop sending false signals!

So why prefer one theory over the other? Well, I'm not surprised that it's mostly economists who support the Cynic's Theory. Economists are understandably interested in revealed preferences3, because revealed preferences are revealed by economic transactions and are the ones that determine the economy. It's perfectly reasonable for an economist to care only about those and dimiss any other kind of preference as a red herring that has to be removed before economic calculations can be done. Someone like a philosopher, who is more interested in thought and the mind, might be more susceptible to the identify-with-conscious-thought Naive Theory.

But notice how the theory you choose also has serious political implications4. Consider how each of the two ways of looking at the problem would treat this example:

A wealthy liberal is a member of many environmental organizations, and wants taxes to go up to pay for better conservation programs. However, she can't bring herself to give up her gas-guzzling SUV, and is usually too lazy to sort all her trash for recycling.

I myself throw my support squarely behind the Naive Theory. Conscious minds are potentially rational5, informed by morality, and qualia-laden. Unconscious minds aren't, so who cares what they think?

 

Footnotes:

1: Caplan says that the lack of interest in Stickk offers support for the Cynic's Theory, but I don't see why it should, unless we believe the mental balance of power should be different when deciding whether to use Stickk than when deciding whether to do anything else.

Caplan also suggests in another article that he has never experienced procrastination as akrasia. Although I find this surprising, I don't find it absolutely impossible to believe. His mind may either be exceptionally well-integrated, or it may send signals differently. It seems within the range of normal human mental variation.

2: Of course, I could be lying here, to signal to you that I have socially acceptable beliefs. I suppose I can only make my point if you often have the same experience, or if you've caught someone else fighting akrasia when they didn't know you were there.

3: Even the term "revealed preferences" imports this value system, as if the act of buying something is a revelation that drives away the mist of the false consciously believed preferences.

4: For a real-world example of a politically-charged conflict surrounding the question of whether we should judge on conscious or unconscious beliefs, see Robin's post Redistribution Isn't About Sympathy and my reply.

5: Differences between the conscious and unconscious mind should usually correspond to differences between the goals of a person and the "goals" of the genome, or else between subgoals important today and subgoals important in the EEA.

Why Real Men Wear Pink

51 Post author: Yvain 06 August 2009 07:39AM

"Fashion is a form of ugliness so intolerable we have to alter it every six months."

-- Oscar Wilde

For the past few decades, I and many other men my age have been locked in a battle with the clothing industry. I want simple, good-looking apparel that covers my nakedness and maybe even makes me look attractive. The clothing industry believes someone my age wants either clothing laced with profanity, clothing that objectifies women, clothing that glorifies alcohol or drug use, or clothing that makes them look like a gangster. And judging by the clothing I see people wearing, on the whole they are right.

I've been working my way through Steven Pinker's How The Mind Works, and reached the part where he quotes approvingly Quentin Bell's theory of fashion. The theory provides a good explanation for why so much clothing seems so deliberately outrageous.

Bell starts by offering his own explanation of the "fashion cycle". He claims that the goal of fashion is to signal status. So far, so obvious. But low-status people would like to subvert the signal. Therefore, the goal of lower class people is to look like upper class people, and the goal of upper class people is to not look like lower class people.

One solution is for the upper class to wear clothing so expensive the lower class could not possibly afford it. This worked for medieval lords and ladies, but nowadays after a while mass production will kick in and K-Mart will have a passable rhinestone based imitation available for $49.95. Once the lower class is wearing the once fashionable item, the upper class wouldn't be caught dead in it. They have to choose a new item of clothing to be the status signal, after a short period of grace the lower class copy that too, and the cycle begins again.

For example, maybe in early 2009 a few very high-status people start wearing purple. Everyone who is "in the know" enough to understand that they are trend-setters switches to purple. Soon it becomes obvious that lots of "in the know" people are wearing purple, and anyone who reads fashion magazines starts stocking up on purple clothing. Soon, only the people too out-of-the-loop to know about purple and the people too poor to immediately replace all their clothes are wearing any other color. In mid-2009, some extremely high-status people now go out on a limb and start wearing green; everyone else is too low-status to be comfortable unilaterally breaking the status quo. Soon everyone switches to green. Wearing purple is a way of broadcasting that you're so dumb or so poor you don't have green clothes yet, which is why it's so mortifying to be caught wearing yesterday's fashion (or so I'm told). When the next cycle comes around, no one will immediately go back to wearing purple, because that would signal that they're unfashionable. But by 2015, that stigma will be gone and purple has a chance to come "back in style".

Bell describes a clever way the rich can avoid immediately being copied by the middle class. What is the greatest fear of the fashionista? To be confused with a person of a lower class. So the rich wear lower class clothes. The theory is that the middle class is terrified of wearing lower class clothes, but the rich are so obviously not lower class that they can get away with it. Bell wrote before the "ghetto look" went into style, but his theory explains quite well why wealthy teenagers and young adults would voluntarily copy the styles of the country's poorest underclass.

Bell also explained a second way to signal high-status: conspicuous outrage. Wear a shirt with the word "FUCK" on it in big letters (or, if you prefer, FCUK). This signals "I am so high status that I think I can wear the word 'FUCK' in big letters on a t-shirt and get away with it." It's a pretty good signal. It signals that you don't give a...well...fcuk what anyone else thinks, and the only people who would be able, either economically or psychologically, to get away with that are the high status1.

The absolute best real world example, which again I think Bell didn't live to see, is the bright pink shirt for men that says "REAL MEN WEAR PINK". The signal is that this guy is so confident in his masculinity that he can go around wearing a pink shirt. It's an odd case because it gets away with explaining exactly what signal it's projecting right on the shirt. And it only works because real men do not wear pink without a disclaimer explaining that they are only wearing pink to signal that they are real men.

Pinker notes the similarity to evolutionary strategies that signal fitness by handicapping. A peacock's tail is a way of signalling that its owner is so fit it can afford to have a big maladaptive tail on it and still survive, just as a rich guy in a backwards baseball cap is signalling that its owner is so rich he can afford to copy the lower class and still get invited to parties. The same process produces a body part of astounding beauty in the animal kingdom, and ghetto fashion in human society. I wonder if nature is laughing at us.

Footnotes:

1: Bell (or possibly Pinker, it's not clear) has a similar theory about art. Buying a hip "modern art" painting that's just a white canvas with a black line through it is supposed to signal "I am so rich that I can afford to pay lots of money for a painting even if it is unpopular and hard to appreciate," or even "I am so self-confident in my culturedness that I can endorse this art that is low quality by all previous standards, and people will continue to respect me and my judgments." Then the middle class starts buying white canvases with black lines through them, and rich people have to buy sculptures made of human dung just to keep up.

Comment author: Yvain 08 August 2009 08:23:03AM 11 points [-]

This writer of this essay (seen on Reddit) is a true practical rationalist and role model for all of us.

It's not just because she made a good decision and didn't get emotionally worked up. She was able to look behind the human level and all its status and blame games, see her husband as a victim of impersonal mental forces (don't know if she knows evo psych, but she certainly has an intuitive grasp of some of the consequences) , and use her understanding to get what she wants. And she does it not in a manipulative or evil kind of way, but out of love and a desire to hold her family together.

Comment author: Yvain 07 August 2009 05:45:50AM 10 points [-]

The best time to plant a tree is thirty years ago; the second best time is now.

-- Eliezer Yudkowsky

Comment author: Yvain 07 August 2009 05:44:55AM *  3 points [-]

WARNING: Beware of things that are fun to argue

-- Eliezer Yudkowsky

Comment author: Yvain 07 August 2009 05:44:20AM 5 points [-]

It is a more inspiring battle cry to scream, "Die, vicious scum!" instead of "Die, people who could have been just like me but grew up in a different environment!" 

-- Eliezer Yudkowsky

Comment author: Yvain 07 August 2009 05:42:51AM 4 points [-]

One possible way out [of the Taiwan issue] would be to put a time limit on our defense of Taiwan. Say, 75 years. Then we could say that everyone born in Taiwan today could expect to live out their lives under US protection - but after that, they would have to look out for themselves. This might suit both nations' time horizons. To the Chinese, 75 years might seem like a long wait. To Americans, 50 years is actually longer than "never". You can verify this experimentally: Ask Americans whether something (fusion power, artificial intelligence, a cure for aging, a Best Picture for an animated movie) will ever happen, or whether it will happen in 50 years. More will say that it will happen in 50 years than that it will ever happen.

-- Phil Goetz

Why Real Men Wear Pink

51 Post author: Yvain 06 August 2009 07:39AM

"Fashion is a form of ugliness so intolerable we have to alter it every six months."

-- Oscar Wilde

For the past few decades, I and many other men my age have been locked in a battle with the clothing industry. I want simple, good-looking apparel that covers my nakedness and maybe even makes me look attractive. The clothing industry believes someone my age wants either clothing laced with profanity, clothing that objectifies women, clothing that glorifies alcohol or drug use, or clothing that makes them look like a gangster. And judging by the clothing I see people wearing, on the whole they are right.

I've been working my way through Steven Pinker's How The Mind Works, and reached the part where he quotes approvingly Quentin Bell's theory of fashion. The theory provides a good explanation for why so much clothing seems so deliberately outrageous.

Bell starts by offering his own explanation of the "fashion cycle". He claims that the goal of fashion is to signal status. So far, so obvious. But low-status people would like to subvert the signal. Therefore, the goal of lower class people is to look like upper class people, and the goal of upper class people is to not look like lower class people.

One solution is for the upper class to wear clothing so expensive the lower class could not possibly afford it. This worked for medieval lords and ladies, but nowadays after a while mass production will kick in and K-Mart will have a passable rhinestone based imitation available for $49.95. Once the lower class is wearing the once fashionable item, the upper class wouldn't be caught dead in it. They have to choose a new item of clothing to be the status signal, after a short period of grace the lower class copy that too, and the cycle begins again.

For example, maybe in early 2009 a few very high-status people start wearing purple. Everyone who is "in the know" enough to understand that they are trend-setters switches to purple. Soon it becomes obvious that lots of "in the know" people are wearing purple, and anyone who reads fashion magazines starts stocking up on purple clothing. Soon, only the people too out-of-the-loop to know about purple and the people too poor to immediately replace all their clothes are wearing any other color. In mid-2009, some extremely high-status people now go out on a limb and start wearing green; everyone else is too low-status to be comfortable unilaterally breaking the status quo. Soon everyone switches to green. Wearing purple is a way of broadcasting that you're so dumb or so poor you don't have green clothes yet, which is why it's so mortifying to be caught wearing yesterday's fashion (or so I'm told). When the next cycle comes around, no one will immediately go back to wearing purple, because that would signal that they're unfashionable. But by 2015, that stigma will be gone and purple has a chance to come "back in style".

Bell describes a clever way the rich can avoid immediately being copied by the middle class. What is the greatest fear of the fashionista? To be confused with a person of a lower class. So the rich wear lower class clothes. The theory is that the middle class is terrified of wearing lower class clothes, but the rich are so obviously not lower class that they can get away with it. Bell wrote before the "ghetto look" went into style, but his theory explains quite well why wealthy teenagers and young adults would voluntarily copy the styles of the country's poorest underclass.

Bell also explained a second way to signal high-status: conspicuous outrage. Wear a shirt with the word "FUCK" on it in big letters (or, if you prefer, FCUK). This signals "I am so high status that I think I can wear the word 'FUCK' in big letters on a t-shirt and get away with it." It's a pretty good signal. It signals that you don't give a...well...fcuk what anyone else thinks, and the only people who would be able, either economically or psychologically, to get away with that are the high status1.

The absolute best real world example, which again I think Bell didn't live to see, is the bright pink shirt for men that says "REAL MEN WEAR PINK". The signal is that this guy is so confident in his masculinity that he can go around wearing a pink shirt. It's an odd case because it gets away with explaining exactly what signal it's projecting right on the shirt. And it only works because real men do not wear pink without a disclaimer explaining that they are only wearing pink to signal that they are real men.

Pinker notes the similarity to evolutionary strategies that signal fitness by handicapping. A peacock's tail is a way of signalling that its owner is so fit it can afford to have a big maladaptive tail on it and still survive, just as a rich guy in a backwards baseball cap is signalling that its owner is so rich he can afford to copy the lower class and still get invited to parties. The same process produces a body part of astounding beauty in the animal kingdom, and ghetto fashion in human society. I wonder if nature is laughing at us.

Footnotes:

1: Bell (or possibly Pinker, it's not clear) has a similar theory about art. Buying a hip "modern art" painting that's just a white canvas with a black line through it is supposed to signal "I am so rich that I can afford to pay lots of money for a painting even if it is unpopular and hard to appreciate," or even "I am so self-confident in my culturedness that I can endorse this art that is low quality by all previous standards, and people will continue to respect me and my judgments." Then the middle class starts buying white canvases with black lines through them, and rich people have to buy sculptures made of human dung just to keep up.

Comment author: Yvain 06 August 2009 05:57:38AM *  11 points [-]

[Mathematical methods of inference] literally have no content; long division can calculate miles per gallon, or it can calculate income per capita. The statistical tools of experimental psychology were borrowed from agronomy, where they were invented to gauge the effects of different fertilizers on crop yields. The tools work just fine in psychology, even though, as one psychological statistician wrote, "we do not deal in manure, at least not knowingly."

-- Steven Pinker, How The Mind Works

Comment author: pjeby 04 August 2009 10:52:58PM 3 points [-]

Can you give some concrete examples of people who you think are making this mistake?

He already did:

A correlation of 0.6 is generally considered pretty high in psychology and social science

This is a standard PCT criticism of psychology and social science, i.e., that these low correlation levels are an indicator that they're measuring the wrong things, and indeed using the wrong model of what kinds of things to measure.

(Specifically, the complaint is that those sciences assume organisms are open-loop stimulus-response reactors rather than closed-loop controllers, so they try to measure input-output correlations in their experiments, instead of using their experiments to identify the variables that organisms are able to control or tend to control.)

Comment author: Yvain 05 August 2009 02:42:43AM *  6 points [-]

But social science doesn't respect a correlation of .6 because they think it's a good way to measure something that could be measured directly. They find correlations either as an important step in establishing causation, a way to get large-scale trends, or a good way to measure something that can't be measured directly.

The correlation between smoking and lung cancer is only .7, but that's a very interesting fact. True, just picking out smokers is a terrible way to predict who has lung cancer when compared to even a so-so screening test, which is what I interpreted the point of Richard's article as being. But knowing that there's a high correlation there is useful for other reasons. Since we now know it's causative, we can use it to convince people not to smoke. Even if we didn't know there was causation, it would at least help us to pick out who needs more frequent lung cancer screening tests.. So I am not prepared to immediately accept that someone is doing something wrong if they call a correlation of .6 pretty high.

Can you or Richard give an example of something the people investigating lung cancer could have done with direct measurement that would have been more productive than analyzing the cigarettes-smoking correlation? If not, can you provide a situation where people did overuse correlations when they'd have been better off using a measurement?

Comment author: Yvain 04 August 2009 08:57:02PM 4 points [-]

Can you give some concrete examples of people who you think are making this mistake?

Comment author: thomblake 04 August 2009 08:20:05PM 2 points [-]

I think it's obvious which would be more right. But the real question is, which would be more prime?

Comment author: Yvain 04 August 2009 08:52:35PM *  3 points [-]

No, I'm asking which would be more right, and I don't think it's obvious.

If I like steak and you're a vegetarian, and I'm feeling altruistic according to normal human standards, which do I give you? Steak or vegetables?

My utility function contains a term that sort of resembles a desire to increase your utility function, which is why I'd probably give you vegetables. But intuitively I feel like this only works when your utility function is close enough to mine that I can at least sympathize with it. I honestly don't know what I'd do in the Pebble Sorter situation.

Comment author: Tom_Talbot 02 August 2009 05:41:24PM 4 points [-]

Imagine you find a magic lamp. You polish it and, as expected, a genie pops out. However, it's a special kind of genie and instead of offering you three wishes it offers to make you an expert in anything, equal to the greatest mind working in that field today, instantly and with no effort on your part. You only get to choose one subject area, with "subject area" defined as anything offered as a degree by a respectable university. Also if you try to trick the genie he'll kick you in the nads*.

So if you could learn anything, what would you learn?

*This example is in no way intended to imply that women are less worthy of the right to be attacked by genies. Neither is it intended to imply that there could never be a female genie. That would be stupid. Where else could baby genies come from?

Comment author: Yvain 04 August 2009 07:40:51PM 0 points [-]

For personal interest, neuroscience (and the genie would wave his wand, and I would be V. Ramachandran). For benefit to society, probably genetics (or do colleges offer degrees in AI?)

I'd also like to see if I could use the genie to answer one of the great questions of the ages. I guess it all depends on how the "expert" thing is implemented. For example, if the genie created a great expert in quantum mechanics, would the expert simply know and understand facts about quantum mechanics, or would they also be such an expert as to have the correct opinion on the Copenhagen vs. many worlds question? After all, Tom did say "an expert equivalent to the greatest mind today", and there are minds that are pretty sure they know the answer to that question, so the mind that has the correct opinion on it must be greater than an equivalent mind that doesn't. That means if I wake up and find myself believing Many Worlds, I have very strong evidence that Many Worlds is correct.

If I thought that plan would work, I'd probably choose Philosophy. I might get kicked in the 'nads, but for the chance to have genie-approved answers all the great philosophical questions at once, it'd be worth it.

Comment author: Yvain 04 August 2009 07:24:03PM *  0 points [-]

Due to an unfortunate accident with a particle accelerator, you are transformed into a mid-level deity and the rest of the human race is wiped out. Experimenting with your divinity, you find you have impressive though limited levels of control over the world, but not enough finesse or knowledge to rewire the minds of intelligent creatures or create new ones.

With humanity gone, you discover that the only intelligent race left in the universe is the Pebble Sorters.

Do you use your newfound powers to feed starving Pebblesorters, free their slaves, slay their tyrants, heal their sick, preserve their places of natural beauty, protect their rights, and end their wars? Or do you use it to build lots and lots of heaps of prime-numbered heaps of pebbles?

Which course of action do you think would be more moral?

In response to comment by Yvain on Suffering
Comment author: teageegeepea 04 August 2009 02:37:34AM 6 points [-]

I suspect there are a higher-than-average number of bullet-biters here, and I number myself among them. I don't grant the intuitions which lead people to dodge bullets much credence.

Although I am a gun-owner, I don't think I am substantially more likely to shoot anyone (delicious animals are another story) than the others here. Though you may think my above-mentioned criteria (including the government as a source of ass-kicking and taking into account risk aversion) don't count, I'd say that constitutes a substantial unwillingness. Also, while this is pedantic, I'd like to again emphasize the importance of disease over guns. Note that north america and australia have had nearly complete population replacement by europeans, while africa has been decolonized. The reason for that is not technology, but relative vulnerability to disease.

If it makes you feel any better about the inhabitants of Less Wrong, note that your reaction was voted up while my response (which was relevant and informative with links to more information, if I may judge my own case for a moment) was voted down. I do not say this to object to anyone's actions (I don't bother voting myself and have no plans to make a front-page post) but to indicate that this is evidence of what the community approves.

Although, as mentioned, I don't believe in objective normative truth, we can pretend for a little while in response to joeteicher. We believe we have a better understanding of many things than 1700s colonists did. If we could bring them in a time-machine to the present we could presumably convince them of many of those things. Do you think we could convince them of our moral superiority? From a Bayesian perspective (I think this is Aumann's specialty) do they have any less justification for dismissing our time period's (or country's) morality as being obviously wrong? Or would they be horrified and make a note to lock up anyone who promotes such crazy ideas in their own day?

G. K. Chesterton once said tradition is a democracy in which the dead get to vote (perhaps he didn't know much about Chicago), which would certainly not be a suitable mechanism of electing representatives but gets to an interesting point in majoritarian epistemology. There are simply huge numbers of people who lived in the past and had such beliefs. What evidence ancient morality?

In response to comment by teageegeepea on Suffering
Comment author: Yvain 04 August 2009 03:22:21AM *  7 points [-]

I don't doubt you're a nonviolent and non-aggressive guy in every day life, nor that in its proper historical context the history of colonists and Indians in the New World was really complicated. I wasn't asking you the question because of an interest in 18th century history, I was asking it as a simplified way to see how far you were taking this "Anyone who can't kick ass isn't a morally significant agent" thing.

Your willingness to take it as far as you do is...well, I'll be honest. To me it's weird, especially since you describe yourself as an emotivist and therefore willing to link morality to feeling. I can think of two interpretations. One, you literally wouldn't feel bad about killing people, as long as they're defenseless. This would make you a psychopath by the technical definition, the one where you simply lack the moral feelings the rest of us take for granted. Two, you have the same tendency to feel bad about actually killing an Indian or any other defenseless person as the rest of us, but you want to uncouple your feelings from "rationality" and make a theory of morality that ignores them (but then how are you an emotivist?!). I know you read all of the morality stuff on Overcoming Bias and that that stuff gave what I thought was a pretty good argument for not doing that. Do you have a counterargument?

(Or I could be completely misunderstanding what you're saying and taking your statement much further than you meant for it to go.)

By the way, I didn't downvote your response; you deserve points for consistency.

In response to comment by Yvain on Suffering
Comment author: teageegeepea 03 August 2009 10:43:19PM *  4 points [-]

I am an emotivist and do not believe anything is good or bad in an objective sense. I think some Indians may have had guns by the 1700s, but their bows and arrows weren't terribly outclassed by many of the old muskets back then either (I'm actually discussing that at my blog right now). The biggest advantage of the colonists was their ever-increasing numbers (while disease steadily drained those of the natives). The indians frequently did respond in kind to killings and the extent to which they could do so would strike me as as the most significant factor to take into consideration when it comes to the decision to kill them.

There is also the factor of trade relations that could be disrupted, but most people engaged in prolonged voluntary trade are going to have significant ass-kicking ability or otherwise they would have been conquered and their goods seized by force already. I understand Peter Leeson has a paper "Trading with bandits" disputing that point, but the frequency with which dominance based resource extraction occurs makes me think the phenomena he discusses only occur under very limited conditions.

In response to comment by teageegeepea on Suffering
Comment author: Yvain 03 August 2009 11:29:10PM *  9 points [-]

Well, I can't accuse you of having any unwillingness to bite bullets. Nor of having any unwillingness to do lots of other questionable things with bullets besides.

Still, Less Wrong has got to be the only place where I can ask if it's okay to massacre Indians, and get one person who says it depends what the people living back then thought, and another who says it depends on the sophistication of musketry technology. I don't know if that's a good thing or a bad thing about this site.

In response to Suffering
Comment author: Yvain 03 August 2009 11:23:43PM 3 points [-]

For me, a morally significant agent is one that has positive and negative qualia. Since I can't tell this by looking at an agent, I guess. I'm pretty sure that my brother does, and I'm pretty sure that a rock doesn't.

Cyan mentioned pain asymbolia on the last thread: the ability to feel pain, but not really find anything painful or problematic about it. If someone had asymbolia generalized across all mental functions, I would stop counting that person as a moral agent.

In response to Suffering
Comment author: teageegeepea 03 August 2009 05:45:06PM *  0 points [-]

BTW, I think the most important defining feature of an MSA is ability to kick people's asses. Very humanizing.

I don't know if you meant that as a joke, but that's pretty much my take from a contractarian perspective (though I wouldn't use the phrase "morally significant agent"). Fish can't do much about us cooking and eating them, so they are not a party to any social contract. That's also the logic behind my tolerance of infanticide.

In response to comment by teageegeepea on Suffering
Comment author: Yvain 03 August 2009 06:02:57PM 9 points [-]

Was it okay to kill the Indians back in the 1700s, before they got guns? What were they going to do? Throw rocks at us?

In response to comment by Yvain on Pain
Comment author: kpreid 03 August 2009 02:19:12AM *  0 points [-]

Please don't take me as having thought this through thoroughly...

I did not intend for that description to be considered outside the person. All of what you're describing are plans the person or animal themselves would disagree with (if they could), yes?

Here's a different statement of roughly the same idea: "My excess pain is bad because it interferes with what I want to do, without benefiting anyone else."

In response to comment by kpreid on Pain
Comment author: Yvain 03 August 2009 03:07:37AM *  5 points [-]

Maybe I'm misunderstanding what you're doing. Your criteria seems to set a standard for determining which pain is bad, and that criteria I would agree with. The pain that's bad is the pain beyond that necessary to send a useful signal.

What I interpreted Alicorn as asking was why pain is bad in the first place. A lot of things can be useless, for example a tune that keeps playing in your head, but useless pain seems to be worse than useless anything else because of something especially bad about pain. Even from an intrapersonal perspective, I can't agree that pain is all about goals. Consider the following thought experiment:

I offer you two choices for tomorrow. Option one: I will torture you for six hours, using a method that is very painful but will leave no lasting scars or aftereffects, and you can spend the rest of the day doing whatever you want. Option two: I will give you a sedative that causes you to sleep through all of tomorrow: you will wake up the day after tomorrow.

If the only problem with pain was that it interferes with things people want to do, then everyone should take Option 1 without a second thought: the pain interferes with what they want to do for six hours, and then they can spend the rest of the day free. But I would take Option 2 (would you?) suggesting that there is more to the negative value of pain than simple inability to do things while you're experiencing it.

In response to comment by Yvain on Pain
Comment author: Alicorn 02 August 2009 10:23:31PM *  6 points [-]

Pokemons

The plural of Pokémon is Pokémon.

In response to comment by Alicorn on Pain
Comment author: Yvain 02 August 2009 10:24:16PM 9 points [-]

...give me one reason not to break your bones right now.

In response to comment by Yvain on Pain
Comment author: SilasBarta 02 August 2009 09:53:56PM *  0 points [-]

I don't think your counterexamples accurately state a scenario or apply kpreid's reasoning correctly:

There's no reason to alleviate the pain of people who are incapacitated (eg bedridden hospital patients, people undergoing surgery) because they wouldn't be accomplishing any goals anyway.

He was clear that the pain in surgery conveys no new information. Plus, the bedridden can clearly accomplish some goals.

There's no reason to care if a person with no goals or bad goals is in pain.

In this context, it's not really possible for someone not to have goals. They might not explicitly be able to state long-term goals, but as long as they're taking deliberate actions, they have goals. And yes, for sufficiently bad goals, you do care how hard it is for the person to carry them out!

There's no reason to care about low levels of pain that don't prevent someone from achieving a goal.

The standard is hindrance, not prevention, and any level of pain will hinder, or you will not identify it as pain.

In response to comment by SilasBarta on Pain
Comment author: Yvain 02 August 2009 10:20:23PM *  10 points [-]

He was clear that the pain in surgery conveys no new information.

Let me try to explain this better, then. Imagine we take a person who needs surgery but was never told by their doctor what part of their body the surgery will be on. We perform the surgery without any anaesthetic and with the patient blindfolded. In this case, the pain is giving new information ("AAAIE! MY RIGHT LEG!") but we still don't approve.

kpreid could clarify that this information is useless (in that the patient doesn't gain anything from knowing) and that (s)he meant useful information. But this isn't true either. I could state before the surgery that I will give the patient ten cents if they can tell me which of their limbs I operated on, but this still wouldn't make it okay to perform surgery without anaesthetic.

The way I would have put kpreid's point is that the pain must provide sufficiently useful information to offset its painfulness. If putting someone under surgery without anaesthetic earned someone ten cents, I would consider it an atrocity, but if it was necessary so that the patient could help guide the surgeon by telling them what they feel, saving the patient's life, then it might be a necessary measure.

However, this seems like straightforward utilitarianism, in which the benefit of getting information must outweigh the cost of having such terrible pain. This means it can't be used as a definition of why pain is a cost.

I would say that pain is a cost for other reasons, but that when pain conveys information, the information can be a counterbalancing factor. This makes it a mistake to say that the reason pain is a cost is that it doesn't convey information, equivalent to saying that the reason bombing civilians in Afghanistan is bad is that it doesn't kill Osama bin Laden. The reason bombing civilians is bad is because murder is wrong. Killing Osama bin Laden would be a potential counterbalancing factor that might justify bombing the civilians, but lack of Osama-killing is not the definition of the evil of murder.

In this context, it's not really possible for someone not to have goals. They might not explicitly be able to state long-term goals, but as long as they're taking deliberate actions, they have goals. And yes, for sufficiently bad goals, you do care how hard it is for the person to carry them out!

Consider someone whose only ambition is to collect every Pokemon in the world. Kpreid's scenario suggests a dichotomy: either it is okay to cause this person pain, or the only reason not to cause this person pain is because it might prevent Pokemons from being collected. I don't think this captures the reason we don't break the bones of Pokemon collectors (even though we all feel sorely tempted sometimes.)

In response to Pain
Comment author: kpreid 02 August 2009 07:33:36PM 16 points [-]

Pain is sensory input; ideally it conveys useful information to the brain.

Pain which does not convey new information is bad, because it interferes with working towards what one values.

  • The anesthesiologist is removing pain which does not convey new information.
  • The CIPA patient lacks pain whether or not it conveys new information, and therefore lacks information.
In response to comment by kpreid on Pain
Comment author: Yvain 02 August 2009 09:41:00PM *  12 points [-]

If I understand what you're saying, and if you really mean that is the sole or even the main reason why pain is a bad thing, I don't like that line of thought at all. It implies:

  • There's no reason to alleviate the pain of people who are incapacitated (eg bedridden hospital patients, people undergoing surgery) because they wouldn't be accomplishing any goals anyway.
  • There's no reason to care if a person with no goals or bad goals is in pain.
  • There's no reason to care about low levels of pain that don't prevent someone from achieving a goal.
  • Any infliction of pain that increases a person's productivity, for example whipping slaves, has no downside (to one who already accepts slavery).
  • There's no problem with inflicting pain on animals, since they don't have any interesting values anyway.
  • If Hell existed, it wouldn't be a big problem because what are you going to be accomplishing after you die anyway?
  • "Alleviate other people's pain" shouldn't be a supergoal, but only a subgoal depending on whether you like what those people's goals are.

We don't have to work for the sake of happiness alone, but happiness can't be entirely subsumed by other wants.

In response to Pain
Comment author: Yvain 02 August 2009 09:36:49PM *  62 points [-]

This is a form of the general question "What's so bad about X?" with pain as X.

For any X, we can ask "What's so bad about X", receive an answer X2, and ask "What's so bad about X2", ad infinitum. The three most common responses are semantic stopsigns, moral nihilism, and admitting you need to ask the question more rigorously.

Once phrased more rigorously, the problem becomes easier, transforming into some combination of:

"Why do people dislike pain?", to which the answer is that it's hard-wired into the brain in some way a neurologist could probably explain, probably similar to how it's hard-wired to dislike things that taste bitter.

"Why do people call pain bad?", to which the answer is that most people think as emotivists, and call pain bad because they dislike it.

"Why is pain bad in Moral System Y?", to which the answer is that you'd have to ask the people in moral system Y, and they'll give you their moral system's answer. I think a lot of the better moral system would have it as an axiom. They probably make it an axiom because most moral systems are linked in some way or another to what people do or don't like, whether they admit it or not.

"Why is there a strong negative qualia for pain instead of it just feeling like a little voice at the back of your head saying 'that's painful'?", to which the answer will remain mysterious until we understand qualia, but no more mysterious than any other sensation.

Comment author: Psychohistorian 31 July 2009 12:55:15AM *  1 point [-]

The catch is that all of them are illegal, even the ones that can be used safely for the majority of people.

Such as...?

That's an enormous claim to make; I think it's fair to say such a claim should be accompanied by some kind of evidence, as I seriously doubt it's common knowledge if true.

Comment author: Yvain 31 July 2009 01:02:23AM 0 points [-]

Olestra (not a drug, but a chemical solution to part of the obesity problem) had a lot of promise. It didn't get banned, but people did freak out about a generally minor side effect, hold a health scare, and finally send it into commercial oblivion.

In response to comment by dial911 on The Obesity Myth
Comment author: cousin_it 30 July 2009 02:21:47PM *  0 points [-]

The American health care system wastes billions of dollars each year on diseases and conditions that are a direct cause of obesity.

I assume you meant "a direct consequence of obesity". Here's a quote from the original interview that contradicts your claim:

Paul: The correlations between higher weight and greater health risk are weak except at statistical extremes. The extent to which those correlations are causal is poorly established. There is literally not a shred of evidence that turning fat people into thin people improves their health. And the reason there's no evidence is that there's no way to do it.

Now, if your point of view is informed, the right way forward in the argument would be producing citations.

Also, the part of your comment that deals with evolution seems confused to me. Hunter-gatherers don't get obese because they don't have access to cheap food, not because natural selection kills off obese people. If the latter were the case, modern people (who evolved from hunter-gatherers, after all) would have genetic safeguards against obesity.

In response to comment by cousin_it on The Obesity Myth
Comment author: Yvain 31 July 2009 12:13:06AM *  5 points [-]

Now, if your point of view is informed, the right way forward in the argument would be producing citations.

I am literally so baffled by the thesis presented above I can't bring myself to disagree with it because I must be missing something. I mean, within five minutes on Google I found five studies showing strong correlations between various measures of body fat and coronary heart disease:

In a 14-year prospective study, middle-aged women with a BMI >23 but <25 had a 50% increase in risk of nonfatal or fatal coronary heart disease,9 and men aged 40 to 65 years with a BMI >25 but <29 had a 72% increased risk

When participants with the highest waist-to-hip ratio were compared with those having the lowest ratio, there was an 80 percent increase in risk.

The risk of any CHD event, a nonfatal event, and a fatal event among adults was positively associated with BMI at 7 to 13 years of age for boys and 10 to 13 years of age for girls.

These prospective cohort studies employed simple indices of body-fat distribution such as waist-to-hip circumference ratio or subscapular skinfold. Their similar results suggested that increased abdominal obesity conferred a two-fold increased risk of ischemic heart disease among middle-aged men.

IAF and trunk fat were consistently positively related to CVD risk factors, whereas leg fat was consistently negatively related to CVD risk, indicating that IAF and trunk fat may put women at increased risk of developing CVD.

If I was willing to give it an hour, I'm sure I could have found sixty of them, and I know there are many similar studies about type II diabetes, cerebrovascular disease, et cetera. Because many of these are prospective studies, they have a better ability to show causality than longitudinal studies (although still not perfect). And another five minutes on Google find me several interventional studies about how turning fat people into thin people improves their health:

Our study has shown that weight loss of as little as 6.5 percent in individuals with [metabolic syndrome] results in substantial reductions in blood pressure, glucose, triglycerides and total cholesterol, all factors that lead to heart disease,’ MDBHVC director Christie Ballantyne said. These impressive results occur early in the weight loss, well before individuals even begin to approach their ideal body weight,’ Ballantyne said.

Weight loss was significantly associated with lower rate of the composite outcome after adjustment for age, sex, smoking, dyslipidemia, diabetes, hypertension, myocardial infarction, and obese status [hazard ratio (HR)=0.62; P=0.018]. Subgroup analysis showed that patients who lost weight had favorable outcomes both in patients with BMI <25 (HR=0.32; P=0.035) and those with BMI ≥25kg/m2 HR=0.64; P=0.032.

So what exactly is the thesis? That all of these studies are flawed in the same way? That there's some vital causal step that's been left out? Surely the author must know about these, right?

Comment author: Wei_Dai 24 July 2009 06:56:22AM *  9 points [-]

Your instances do not include politically incorrect statements (racist, sexist, or various other -ists, depending on who exactly is listening), whether factually incorrect or otherwise, which seem to be one of if not the major sources of serious offense.

A racist statement is usually one that, if accepted by the listener, will tend to lower the status of the targeted race. Same for other -ists. I'm not seeing how it doesn't fit with the status theory.

If I ran into a church picnic and started yelling obscenities, people would get offended, even though I'm not threatening their "high" status so much as advertising my "low" status.

Of course you're threatening their high status. You're implying that vulgar language is appropriate in their social circle, and the only way it could be appropriate is if they have low status.

you can make incredibly offensive comments towards a close friend with no ill effect, yet a person hearing the exchange might themselves be offended

Edit: On second thought, I think what's going on here is that once you're a close enough friend with someone, there is no longer a significant chance that you'd want to intentionally lower their status, so an otherwise offensive comment (especially in private) becomes a signal for close friendship. You're signaling that you believe your friendship is so close that your friend won't think you're intending harm, and by not taking offense, your friend then signals the same thing. This probably takes a mathematical model to make completely clear, but maybe you get the gist.

Comment author: Yvain 25 July 2009 07:03:23AM 8 points [-]

Racist statements don't seem to automatically imply lower status for the offended group. For example, many people found this "joke" offensive, even though the only claim seems to be that black people eat lots of watermelon. Similarly, a statement like "Jews control the financial system" could easily offend Jews, even though if anything it assigns them high status.

In response to Many Reasons
Comment author: Yvain 25 July 2009 05:31:18AM 1 point [-]

I don't understand what you're saying. The law of conservation of expected evidence applies only to expectations of evidence and is conditional upon your current level of belief.

It seems to me that your possibility that "what you're describing as many reasons is really a set of different manifestations of the same underlying reason" is exactly the point at issue. The underlying reason is that something is true. I can think of many reasons to believe Russia exists. I've seen pictures of it, I know people who have been there, I've read literature by Russian writers, et cetera. I know of no evidence that Russia does not exist. I can think of more reasons to believe Russia exists than that Russia does not exist because Russia exists. Likewise, I can think of many more reasons not to kill a policeman than I can think of reasons to kill a policeman, and this is because killing a policeman is a bad idea.

Am I missing your point?

Comment author: Eliezer_Yudkowsky 23 July 2009 05:15:31PM 20 points [-]

I would still be better off just giving the beggar the money!

Increasing the payoff of begging just increases the number of beggars until the average wage once again falls to the point where it is just marginally unattractive - or so I've heard. I think someone once suggested that if you visit a poor country and want to help, do not under any circumstances give money to beggars; find someone who seems to be doing something productive, and give the money to them.

Comment author: Yvain 23 July 2009 07:47:01PM *  9 points [-]

...no, I don't see why that means one shouldn't give money to beggars.

Let's say beggars in my city can make $1/hour. Everyone who, in a world without begging, would have made <$1/hour becomes a beggar, and everyone who in a world without begging would have made >$1 hour does not become a beggar.

Now let's say people in my city become more generous and give more money. The number of beggars cannot increase without the wage of beggars also increasing, because the only reason more people would become beggars is that there is a higher incentive, and if the influx of new beggars drive wages back down, those new beggars will "quit" their "job" and the hourly wage will stabilize. So it may be that now beggars earn $2/hour, and everyone who would earn <$2/hour at normal labor quits and becomes a beggar.

In a country with a minimum wage, this suggests that no one will ever leave a job for begging until a beggar's wage exceeds minimum wage; despite the horror stories I don't think this has happened here. It suggests that the population of beggars will probably consist of the people who would have a (low-paying) job if there was no minimum wage but can't get any job in the current regulatory climate. These people seem worth helping.

In a country with no minimum wage, begging establishes an effective minimum wage. That is, if beggars can earn $1/day, then no one will work for less than $1/day because they'd rather beg. This may be bad from an economic standpoint, but it's good from a humanitarian standpoint; it means we can be assured every poor person in the country will earn at least $1/day, whether working or begging, and that no one will have to make do with less. If people become more generous and donations rise to $2/day, this just means that all poor people can be assured of a little bit more money. This seems like exactly what people donating to beggars have as their goal.

Although it may leave a bad taste in our mouths that beggars are quitting their fifty-cent-a-day jobs for begging, if we place a greater value on people not having to live on fifty cents a day than we do on the "moral value of hard work", we are making people better off. We'd have to balance that against the lost productivity of these beggars' fifty-cent-a-day jobs,but if they're only earning fifty cents a day, they can't produce all that much.

[possible counter-argument: if you have a fifty-cent-a-day job, you might learn skills and get promoted. This seems a much more relevant concern in a first world country than in a third world country, where most people labor at dead-end jobs their whole lives, and since first world countries generally have minimum wages anyway, I don't consider it too important]

[this argument probably only works in Economics Land; I think gworley does a better job of describing what happens in the real world.]

Comment author: Yvain 23 July 2009 12:17:54PM 6 points [-]

IAWYC and your examples reflect my own experience.

But unless there's some difference between the amount of planning, thinking, and daydreaming necessary for our ancestral environment and the amount of those things necessary now, evolutionary psychology at least provides some weak evidence that humans on average plan, think, and daydream about the right amount. That suggests that maybe the costs to asking people on dates and getting stuff done is balanced by benefits elsewhere.

Comment author: orthonormal 22 July 2009 10:23:24PM *  11 points [-]

First off, welcome to Less Wrong! Check out the welcome thread if you haven't already.

You have a good writing style, but I hope you'll pardon me if I make a few suggestions based on the usual audience for Less Wrong posts:

Typically, a post of this length should be broken up into a sequence; you run the risk of "too long; didn't read" reactions after 1000 words, let alone 3000, and the conversation in the comments is usually sharper if the post has a single narrow focus. Usually, the analysis of a situation and the recommendations become separate posts if both are substantial.

Secondly, with the notable exception (sometimes) of P.J. Eby, we're often mistrustful of theories borne of introspection and anecdotes, and especially of recommendations based on such theories. There's therefore a norm of looking for and linking to experimental confirmation where it exists, and being doubly cautious if it doesn't. In this case, for instance, you could find some experimental evidence on choking that supports your thesis. This also forces you to think carefully about what sort of things your model predicts and doesn't predict, since at first glance it seems vague to the point of danger. The more specific you can get about these phenomena, the more useful your post will be.

Comment author: Yvain 23 July 2009 12:13:09PM *  12 points [-]

Although I agree that a theory born of empirical evidence is better than one born of introspection, I think it is kind of dangerous to introspect, develop a theory, and then when you're posting it on Less Wrong look for some evidence to support it so that you can say it's empirical. It risks reducing The Procurement of Evidence to a ritual.

See, the problem is, he could probably tie the evidence about choking into his theory. But if he had the opposite theory, he could probably tie studies like the ones showing mental practice can improve sports performance and the one showing that problem-solving areas of the brain are highly active when we daydream in to support that. That means that the fact that he can find a tangentially related study doesn't really make it more likely that the post is true. It'd just make us feel all nice and empirical

The matter would be different if there happened to be a study about this exact topic, or if there had been some study that had inspired him to come up with this theory. But "come up with theory, find supporting evidence" seems dangerous to me.

AndrewH's observation and opportunity costs

22 Post author: Yvain 23 July 2009 11:36AM

In his discussion of "cryocrastination", AndrewH makes a pretty good point. There may be some better things you can do with the money you'd spend on cryonics insurance. The sort of people who are into cryonics would probably accept that donating it to the Singularity Institute is probably, all in all, a higher utility use of however many dollars. Andrew's conclusion is that you should figure out what maximizes utility and do it, regardless of how small a contribution is involved. He's right, but I want to use the same example to push a point that is very slightly different, or maybe a little more general, or maybe the exact same one but phrased differently.

Consider an argument frequently made when politicians are discussing the budget. I frequently hear people say it would cost between ten and twenty billion dollars a year to feed all the hungry people in the world. I don't know if that's true or not, and considering the recent skepticism about aid it probably isn't, but let's say the politicians believe it. So when they look at (for example) NASA's budget of fifteen billion dollars, they say something like "It's criminal to be spending all this money on space probes and radio telescopes when it could eliminate world hunger, so let's cut NASA's budget."

You see the problem? When we cut NASA's budget, it doesn't immediately go into the "solve world hunger" fund. It goes into the rest of the budget, and probably gets divided among the Congressman Johnson Memorial Fisheries Museum and purchasing twelve-thousand-dollar staplers.

The same is true of cryocrastination. Unless you actually take that money you would have spent on cryonics and donate it to the Singularity Institute, it's going into the rest of your budget, and you'll probably spend it on coffee and plasma TVs and famous statistician trading cards and whatever else.

I find myself frequently making this error in the following way: a beggar asks me for money, and I want to give it to them on the grounds that they have activated my urge to help people. Then think to myself "I can't justify giving the money to this beggar when it would help many more people if I gave it to a responsible charity." So I say no, and forget all about it, and never give the money to anyone. Even though (from a charity point of view) I know of a superior alternative to giving the money to the beggar, I would still be better off just giving the beggar the money!

All this means that for any entity that does not use its resources with maximum efficiency, the opportunity cost of spending a certain amount of resources should not be calculated as what you'd get earn from the best possible use of those resources, but what you'll earn from the use of those resources which you expect to actually occur.

Missing the Trees for the Forest

64 Post author: Yvain 22 July 2009 03:23AM

Politics is the mind-killer. A while back, I gave an example: the government's request that Kelloggs  [EDIT: General Mills, thanks CronoDAS] top making false claims about Cheerios. By the time the right-wing and left-wing blogospheres had finished with it, this became everything from part of the deliberate strangulation of the American entrepreneurial spirit by a conspiracy of bureaucrats, to a symbol of the radicalization of the political right into a fringe group obsessed with Communism, to a prelude to Obama's plan to commit genocide against all citizens who disagree with him. All because of Cheerios!

Why? What drives someone to hear about a reasonable change in cereal advertising policy and immediately think of a second Holocaust?

This reminds me of something I used to notice when reading about politics. Sometimes there would be a seemingly good idea to deregulate something that clearly needed deregulation. The idea's proponents would go on TV and say that, hey, this was obviously a good idea. Whoever by the vagary of politics had to oppose the idea would go on TV and talk about industry's plot to emasculate government safeguards. Predatory corporations! Class solidarity! Consumer safety!

Then the next day, there would be seemingly good idea to regulate something that clearly needed regulating. The idea's proponents would go on TV and say that, hey, this was obviously a good idea. Its opponents would go on TV and say that all government regulation was inherently bad. Small government! Freedom! Capitalism!

I have found a pattern: when people consider an idea in isolation, they tend to make good decisions. When they consider an idea a symbol of a vast overarching narrative, they tend to make very bad decisions.

Let me offer another example.

A white man is accused of a violent attack on a black woman. In isolation, well, either he did it or he didn't, and without any more facts there's no use discussing it.

But what if this accusation is viewed as a symbol? What if you have been saying for years that racism and sexism are endemic in this country, and that whites and males are constantly abusing blacks and females, and they're always getting away with it because the police are part of a good ole' boys network who protect their fellow privileged whites?

Well, right now, you'll probably still ask for the evidence. But if I gave you some evidence, and it was complicated, you'd probably interpret it in favor of the white man's guilt. The heart has its reasons that reasons know not of, and most of them suck. We make unconsciously make decisions based on our own self-interest and what makes us angry or happy, and then later we find reasons why the evidence supports them. If I have a strong interest in a narrative of racism, then I will interpret the evidence to support accusations of racism.

Lest I sound like I'm picking on the politically correct, I've seen scores of people with the opposite narrative. You know, political correctness has grown rampant in our society, women and minorities have been elevated to a status where they can do no wrong, the liberal intelligentsia always tries to pin everything on the white male. When the person with this narrative hears the evidence in this case, they may be more likely to believe the white man - especially if they'd just listened to their aforementioned counterpart give their speech about how this proves the racist and sexist tendencies of white men.

Yes, I'm thinking of the Duke lacrosse case.

The problem here is that there are two different questions here: whether this particular white male attacked this particular black woman, and whether our society is racist or "reverse racist". The first question definitely has one correct answer which while difficult to ascertain is philosophically simple, whereas the second question is meaningless, in the same technical sense that "Islam is a religion of peace" is meaningless. People are conflating these two questions, and acting as if the answer to the second determines the answer to the first.

Which is all nice and well unless you're one of the people involved in the case, in which case you really don't care about which races are or are not privileged in our society as much as you care about not being thrown in jail for a crime you didn't commit, or about having your attacker brought to justice.

I think this is the driving force behind a lot of politics. Let's say we are considering a law mandating businesses to lower their pollution levels. So far as I understand economics, the best decision-making strategy is to estimate how much pollution is costing the population, how much cutting pollution would cost business, and if there's a net profit, pass the law. Of course it's more complicated, but this seems like a reasonable start.

What actually happens? One side hears the word "pollution" and starts thinking of hundreds of times when beautiful pristine forests were cut down in the name of corporate greed. This links into other narratives about corporate greed, like how corporations are oppressing their workers in sweatshops in third world countries, and since corporate executives are usually white and third world workers usually not, let's add racism into the mix. So this turns into one particular battle in the war between All That Is Right And Good and Corporate Greed That Destroys Rainforests And Oppresses Workers And Is Probably Racist.

The other side hears the words "law mandating businesses" and starts thinking of a long history of governments choking off profitable industry to satisfy the needs of the moment and their re-election campaign. The demonization of private industry and subsequent attempt to turn to the government for relief is a hallmark of communism, which despite the liberal intelligentsia's love of it killed sixty million people. Now this is a battle in the war between All That Is Right And Good and an unholy combination of Naive Populism and Soviet Russia. This, I think, is part of what happened to the poor Cheerios.

Now, if the economists do their calculations and report that actually the law would cause more harm than good, do you think the warriors against Corporate Greed That Destroys Rainforests And Oppresses Workers And Is Probably Racist are going to say "Oh, okay then" and stand down? In the face of Corporate Greed That Destroys Rainforests And Oppresses Workers And Is Probably Racist?!?1

One more completely hypothetical example. Let's say someone uses language that objectifies women on a blog. Not out of malice or anything, it was just a post on evolutionary psychology, it's easy to write evolutionary psychology in a way that sounds like it's objectifying women, and since obviously no one would objectify women on purpose to insult them it will be clear to everyone that it was just a harmless turn of phrase. Right?

And let's say some feminist comes along and reads this completely innocent phrase about women. Let's say the context is the entire history of gender relations for the past ten thousand years, in which men have usually oppressed women and usually been pretty okay with doing so. And a society that's moving towards not oppressing women and towards treating them as full and equal human beings, but it's still not entirely clear that everyone's on board with this.

This poorly-worded phrase is now a symbol of All Those Chauvinists Who Think Of Women As Ornaments Or Toys Only Good For Sex And Making Babies2. The feminist is unhappy. He or she asks for the phrase to be removed.

Let's say some person who is emphatically not a feminist notices this request for removal. Let's say the context is a society where men are generally portrayed in popular culture as violent bumbling apes who cause all world problems. A culture where women can go on for hours about what boors men are, but any man who says a word about women is immediately branded a sexist pig. A culture where a popular feminist once said that all sex was rape [EDIT: Or not. Apologies for misquote], and many people believed her, one with affirmative action laws mandating that women be hired over equally qualified men, one where you can't say "chairman of the board" without someone calling you sexist and accusing you of taking advantage of your male privilege to ignore male privilege if you disagree.

This request to remove a potentially offensive phrase is now a symbol of All Those Feminists Who Hate Men And Want Them To Feel Guilty All The Time For Vague Reasons. He or she gets angry, and certainly won't remove the offending phrase.

I'm not sure that's what's happening in this case, but I don't think a few poorly worded phrases followed by a polite request to change those poorly worded phrases would have reached five hundred fifty comments divided over four top-level posts if people were just taking it as a request to use slightly different language. In our completely hypothetical example, of course.

I call this mistake "missing the trees for the forest". If you have a specific case you need to judge, judge it separately on its own merits, not the merits of what agendas it promotes or how it fits with emotionally charged narratives3.

 

Footnotes

1: This gets worse once it gets formally organized into political parties. You get people saying something like "How can you, as an atheist, support the war in Iraq?" and thinking it makes perfect sense, because, after all, the war in Iraq is a Republican initiative, and the Republicans are the party of religious conservatives, therefore... Oh, yes, people think like this.

2: Oh, and this answers a question I sometimes hear asked half-seriously on message boards: how come derogatory jokes are okay in some settings but not in others? For example, how come Polish jokes are generally considered okay, but black jokes definitely aren't? Or how come it's considered okay for a black person to make a racist-sounding joke about black people or use the n-word, whereas it's not okay for a white person?

I think the answer is that if I were to make a Polish joke, it would be interpreted as what it is - a joke that needed somebody to play the part of a stupid person to be funny, and Polish people have traditionally served that role. There is no active well-known ongoing context of persecution of Polish people for the joke to symbolize, so it symbolizes nothing but itself and is inert. If I were to tell a joke about black people, even if it was clear that I wasn't actually racist and just thought the joke was funny, then since most people have a very active concept of persecution of black people, my joke would be a symbol of that persecution, and all right-thinking people who oppose that persecution would also probably oppose my joke. 

This leads to the odd conclusion that in a society known to be without racism, no one would mind racist jokes or slurs. In fact, this is confirmed by evidence. Black people are, society generally assumes, above suspicion when it comes to anti-black racism, and therefore black people can use the "n-word" without most people objecting.

This is what led to me developing some of these thoughts. I told a joke which I considered to be making fun of racism. Someone who heard it misinterpreted it and thought it was racist, accused me of racism, spread rumors that I was racist, and generally started a large and complicated campaign to discredit me. After that, I noticed that I was always coming to the defense of people who were accused of racism, and was willing to dismiss practically the entire concept of racism in society as a self-serving attempt at personal gain by minorities, a one hundred eighty degree turn from my previous attitude. Eventually I realized that I was just re-fighting the battle I had to fight after this one joke, and fitting everything to my "sometimes false accusations of racism unfairly harm majority group members and we need to protect against this" narrative. So I stopped. I think.

This also could explain why, contrary to Robin Hanson's hopes, people will never stop using disclaimers. They're ways of saying "I did this action for reasons that do not relate to your narrative; please exclude me from it", and this is not people's default position.

3: One objection could be that the specific case could start a slippery slope, or create a climate in which other things become viewed as more acceptable. In my experience, neither of these matter nearly as much as they would have to to justify the number of times people invoke them.

Comment author: Yvain 23 July 2009 10:19:06AM *  26 points [-]

I disagree with the way you chose to argue this point.

Consider a statement like "Choosing to denounce your country is cowardly". Taboo cowardly and you get a statement like "Choosing to denounce your country is a decision made in order to avoid personal harm." I like the second sentence better, despite it being obvious and not really saying anything interesting. The word "cowardly" slips in an entire value system without anyone noticing, and this value system is precisely the point that needs to be proven here.

The digression to heroism looks like an attempt to create a false dichotomy: either these people are heroes or they are cowards, there's nothing heroic about doing something expedient, therefore they are cowards (or narcissists, or whatever).

In fact, I will play devil's advocate and entertain the opposite recommendation. The important question seems to be what signal a captured soldier praising the enemy gives. The enemy wants it to send two signals: first, that the captive really has realized that the enemy is right and the home country is wrong, and second, that our soldiers are so weak that they will betray our expectation that they remain loyal to the home country.

Let's say that we took your post at face value and created a strong societal norm that captives should never praise the enemy, no matter how much they are threatened. Assuming some captives still break, this is now a disaster. First, we have trouble explaining why our soldiers have betrayed us and started praising the enemy, so the enemy's preferred explanation - that the soldiers have suddenly realized the enemy is superior - becomes more plausible. Second, we are all demoralized the our soldiers were so weak as to betray the societal normal we created. Result: our society is demoralized and unhappy. This is to say nothing of the poor captives, who probably had to be tortured pretty thoroughly before breaking the norm.

Now, let's say we went the opposite direction, and the military instituted a regulation that all captured soldiers must immediately accept any demands the enemy makes to undergo forced conversions or praise their leadership or whatever. Let's even say the Queen or the President or whoever make a televised speech demanding that all soldiers captured in the future do this, for the good of the country. When the captives do so, we have no trouble explaining this: it's because they were forced. And we are not demoralized at all, because the soldiers followed orders.

(For this to work best, we'd have to make sure that people from the enemy country, for example, the Iranian people, knew about this declaration. Otherwise, it would be too easy for the Iranian media to display our soldiers' statements to the populace.)

I think we're already benefiting from this. I, for one, didn't feel the least bit demoralized when I heard some journalists had converted rather than be tortured and killed, because I just expected it as a matter of course. If we were classical Sparta and they'd done the same, we'd probably be facing an existential crisis as a society by now.

I'm not sure this would work in our current society, but it's how I'd organize things in a society of rationalists. And even in our current society, I would discourage anyone from going out of their way to create an expectation that people have to behave heroically. That's just helping the terrorists :P

Comment author: Yvain 23 July 2009 10:37:28AM *  9 points [-]

Actually, you bring up the persecution of the Jews, and this reminds me of a Jewish tradition, the Kol Nidre. It's a ritual performed on the night before Yom Kippur. The congregation absolves themselves of all vows performed over the past year that...aren't supposed to count...although there's some controversy over what exactly that means.

The story I learned in synagogue was that it was developed by the Spanish Jews, who were forced to swear their allegiance to Christianity. They came up with Kol Nidre as a way of establishing a societal norm that these oaths didn't count, so that they could swear as many of them as they wanted without losing their honor or integrity.

I have since learned that this is an oversimplification of the origin of Kol Nidre and that it probably didn't have much to do with the Spanish Inquisition at all. However, it reminds me a bit of some of the concepts I discussed in the Applied Picoeconomics article. Swearing an oath is important because it provides a way to tie your present action to your future actions - sort of like saying "You can trust me when I take an oath, because if I betrayed you now, no one would ever trust me again when I took an oath, and this would cause me severe negative consequences, therefore it is in my own self-interest to follow this oath." This is why the oaths I discussed in Applied Picoeconomics all had some loopholes, so that if it was necessary to break the oath because of extreme conditions, I could break the oath with my honor intact.

Kol Nidre was (at least in the probably false story about it I learned) a way for the Jews to break an oath and keep their honor intact, at least among other Jews - who were probably the relevant community to them. I could break an oath to the King of Spain on Monday, and then make an oath to my business partner Moishe on Tuesday, and Moishe could have complete confidence that I would keep my oath to him, because the only reason I broke my oath to the King of Spain was that I was permitted to do so under Kol Nidre, which would not excuse my breaking of the oath to Moishe.

This expectation that it wasn't important to keep oaths to the Spanish Crown helped the Jewish community survive with both their integrity and their religion, which the Spanish Crown probably thought was impossible. Creating a similar norm that it's okay to praise Iran when you get captured by them would have the same effect, at least in a fully rational community.

Comment author: Yvain 23 July 2009 10:19:06AM *  26 points [-]

I disagree with the way you chose to argue this point.

Consider a statement like "Choosing to denounce your country is cowardly". Taboo cowardly and you get a statement like "Choosing to denounce your country is a decision made in order to avoid personal harm." I like the second sentence better, despite it being obvious and not really saying anything interesting. The word "cowardly" slips in an entire value system without anyone noticing, and this value system is precisely the point that needs to be proven here.

The digression to heroism looks like an attempt to create a false dichotomy: either these people are heroes or they are cowards, there's nothing heroic about doing something expedient, therefore they are cowards (or narcissists, or whatever).

In fact, I will play devil's advocate and entertain the opposite recommendation. The important question seems to be what signal a captured soldier praising the enemy gives. The enemy wants it to send two signals: first, that the captive really has realized that the enemy is right and the home country is wrong, and second, that our soldiers are so weak that they will betray our expectation that they remain loyal to the home country.

Let's say that we took your post at face value and created a strong societal norm that captives should never praise the enemy, no matter how much they are threatened. Assuming some captives still break, this is now a disaster. First, we have trouble explaining why our soldiers have betrayed us and started praising the enemy, so the enemy's preferred explanation - that the soldiers have suddenly realized the enemy is superior - becomes more plausible. Second, we are all demoralized the our soldiers were so weak as to betray the societal normal we created. Result: our society is demoralized and unhappy. This is to say nothing of the poor captives, who probably had to be tortured pretty thoroughly before breaking the norm.

Now, let's say we went the opposite direction, and the military instituted a regulation that all captured soldiers must immediately accept any demands the enemy makes to undergo forced conversions or praise their leadership or whatever. Let's even say the Queen or the President or whoever make a televised speech demanding that all soldiers captured in the future do this, for the good of the country. When the captives do so, we have no trouble explaining this: it's because they were forced. And we are not demoralized at all, because the soldiers followed orders.

(For this to work best, we'd have to make sure that people from the enemy country, for example, the Iranian people, knew about this declaration. Otherwise, it would be too easy for the Iranian media to display our soldiers' statements to the populace.)

I think we're already benefiting from this. I, for one, didn't feel the least bit demoralized when I heard some journalists had converted rather than be tortured and killed, because I just expected it as a matter of course. If we were classical Sparta and they'd done the same, we'd probably be facing an existential crisis as a society by now.

I'm not sure this would work in our current society, but it's how I'd organize things in a society of rationalists. And even in our current society, I would discourage anyone from going out of their way to create an expectation that people have to behave heroically. That's just helping the terrorists :P

Comment author: HalFinney 22 July 2009 06:40:42AM 8 points [-]

I'm afraid I have to take issue with your Cheerios story in the linked comment. You say of the 4% cholesterol lowering claim, "This is false. It is based on a 'study' sponsored by General Mills where subjects took more than half their daily calories from Cheerios (apparently they ate nothing but Cheerios for two of their three daily meals)." You link to http://www.askdeb.com/blog/health/will-cheerios-really-help-lower-your-cholesterol/ but that says nothing about how much Cheerios subjects ate.

I found this article that describes the 1998 Cheerios research that is the foundation for the claim: http://findarticles.com/p/articles/mi_m0813/is_8_32/ai_n15691320/ . It says that participants ate 3 cups of Cheerios per day, while control subjects ate 3 cups of corn flakes. 1 cup of Cheerios is about 100 calories, so 3 cups would be 300, far less than "more than half their daily calories" for any reasonable adult. Further, this article goes on to report that LDL (bad cholesterol) in the Cheerios group fell from 160 to 153, which looks to me like 4%.

Furthermore, my understanding is that the FDA's complaint is not with the accuracy of Cheerios' claim; it is that it is making such a claim at all, even if truthful. The FDA has a lot of rules about what kinds of health benefits products are allowed to make. It is not enough that a claim appears to be correct; the question is the depth and strength of the evidence behind it. For specific health claims, the FDA basically requires full blown clinical trials, as with drugs. According to the LA Times, http://latimesblogs.latimes.com/shopping_blog/2009/05/fda-warns-general-mills-over-cheerios-cholesterol-claims.html , "The FDA allows some health benefits of foods to be advertised but within strict limits. For instance, a company can say that a diet low in saturated fat and high in fiber-rich foods such as fruit, vegetables and whole grains may reduce the risk of heart disease."

There are reasonable questions to be raised about what policy we want to have for regulating health claims. But demonizing Cheerios and General Mills does not facilitate rational discussion of the issues. Saying that their claim is false, and exaggerating the amount of Cheerios which was eaten in their study, only serves to put Cheerios in an unjustifiably bad light.

And BTW I typically eat 600-1000 calories for breakfast, often cold cereal. Sometimes I eat Cheerios but usually I mix two or three different cereals. 300 calories of cold cereal is not difficult for me. The hard part is holding myself back to only eat that much.

Comment author: Yvain 23 July 2009 01:53:54AM 0 points [-]

Hm, I had an article from which I got my numbers, but now I can't find it anymore. I do see several that say three cups of cheerios per day and 450 calories of Cheerios out of a 1900 calorie diet, but I have no idea where I got that "half your total calories" phrase. Possibly I made a mistake and multiplied 4503, when the 450 is already 3150, or possibly copied from an article that did so.

Comment author: Alicorn 22 July 2009 04:18:51AM 10 points [-]

A culture where a popular feminist once said that all sex was rape

Misquote.

Comment author: Yvain 23 July 2009 01:37:57AM 8 points [-]

Corrected with apology, although for the sake of the argument it's only necessary that people think this was said.

Missing the Trees for the Forest

64 Post author: Yvain 22 July 2009 03:23AM

Politics is the mind-killer. A while back, I gave an example: the government's request that Kelloggs  [EDIT: General Mills, thanks CronoDAS] top making false claims about Cheerios. By the time the right-wing and left-wing blogospheres had finished with it, this became everything from part of the deliberate strangulation of the American entrepreneurial spirit by a conspiracy of bureaucrats, to a symbol of the radicalization of the political right into a fringe group obsessed with Communism, to a prelude to Obama's plan to commit genocide against all citizens who disagree with him. All because of Cheerios!

Why? What drives someone to hear about a reasonable change in cereal advertising policy and immediately think of a second Holocaust?

This reminds me of something I used to notice when reading about politics. Sometimes there would be a seemingly good idea to deregulate something that clearly needed deregulation. The idea's proponents would go on TV and say that, hey, this was obviously a good idea. Whoever by the vagary of politics had to oppose the idea would go on TV and talk about industry's plot to emasculate government safeguards. Predatory corporations! Class solidarity! Consumer safety!

Then the next day, there would be seemingly good idea to regulate something that clearly needed regulating. The idea's proponents would go on TV and say that, hey, this was obviously a good idea. Its opponents would go on TV and say that all government regulation was inherently bad. Small government! Freedom! Capitalism!

I have found a pattern: when people consider an idea in isolation, they tend to make good decisions. When they consider an idea a symbol of a vast overarching narrative, they tend to make very bad decisions.

Let me offer another example.

A white man is accused of a violent attack on a black woman. In isolation, well, either he did it or he didn't, and without any more facts there's no use discussing it.

But what if this accusation is viewed as a symbol? What if you have been saying for years that racism and sexism are endemic in this country, and that whites and males are constantly abusing blacks and females, and they're always getting away with it because the police are part of a good ole' boys network who protect their fellow privileged whites?

Well, right now, you'll probably still ask for the evidence. But if I gave you some evidence, and it was complicated, you'd probably interpret it in favor of the white man's guilt. The heart has its reasons that reasons know not of, and most of them suck. We make unconsciously make decisions based on our own self-interest and what makes us angry or happy, and then later we find reasons why the evidence supports them. If I have a strong interest in a narrative of racism, then I will interpret the evidence to support accusations of racism.

Lest I sound like I'm picking on the politically correct, I've seen scores of people with the opposite narrative. You know, political correctness has grown rampant in our society, women and minorities have been elevated to a status where they can do no wrong, the liberal intelligentsia always tries to pin everything on the white male. When the person with this narrative hears the evidence in this case, they may be more likely to believe the white man - especially if they'd just listened to their aforementioned counterpart give their speech about how this proves the racist and sexist tendencies of white men.

Yes, I'm thinking of the Duke lacrosse case.

The problem here is that there are two different questions here: whether this particular white male attacked this particular black woman, and whether our society is racist or "reverse racist". The first question definitely has one correct answer which while difficult to ascertain is philosophically simple, whereas the second question is meaningless, in the same technical sense that "Islam is a religion of peace" is meaningless. People are conflating these two questions, and acting as if the answer to the second determines the answer to the first.

Which is all nice and well unless you're one of the people involved in the case, in which case you really don't care about which races are or are not privileged in our society as much as you care about not being thrown in jail for a crime you didn't commit, or about having your attacker brought to justice.

I think this is the driving force behind a lot of politics. Let's say we are considering a law mandating businesses to lower their pollution levels. So far as I understand economics, the best decision-making strategy is to estimate how much pollution is costing the population, how much cutting pollution would cost business, and if there's a net profit, pass the law. Of course it's more complicated, but this seems like a reasonable start.

What actually happens? One side hears the word "pollution" and starts thinking of hundreds of times when beautiful pristine forests were cut down in the name of corporate greed. This links into other narratives about corporate greed, like how corporations are oppressing their workers in sweatshops in third world countries, and since corporate executives are usually white and third world workers usually not, let's add racism into the mix. So this turns into one particular battle in the war between All That Is Right And Good and Corporate Greed That Destroys Rainforests And Oppresses Workers And Is Probably Racist.

The other side hears the words "law mandating businesses" and starts thinking of a long history of governments choking off profitable industry to satisfy the needs of the moment and their re-election campaign. The demonization of private industry and subsequent attempt to turn to the government for relief is a hallmark of communism, which despite the liberal intelligentsia's love of it killed sixty million people. Now this is a battle in the war between All That Is Right And Good and an unholy combination of Naive Populism and Soviet Russia. This, I think, is part of what happened to the poor Cheerios.

Now, if the economists do their calculations and report that actually the law would cause more harm than good, do you think the warriors against Corporate Greed That Destroys Rainforests And Oppresses Workers And Is Probably Racist are going to say "Oh, okay then" and stand down? In the face of Corporate Greed That Destroys Rainforests And Oppresses Workers And Is Probably Racist?!?1

One more completely hypothetical example. Let's say someone uses language that objectifies women on a blog. Not out of malice or anything, it was just a post on evolutionary psychology, it's easy to write evolutionary psychology in a way that sounds like it's objectifying women, and since obviously no one would objectify women on purpose to insult them it will be clear to everyone that it was just a harmless turn of phrase. Right?

And let's say some feminist comes along and reads this completely innocent phrase about women. Let's say the context is the entire history of gender relations for the past ten thousand years, in which men have usually oppressed women and usually been pretty okay with doing so. And a society that's moving towards not oppressing women and towards treating them as full and equal human beings, but it's still not entirely clear that everyone's on board with this.

This poorly-worded phrase is now a symbol of All Those Chauvinists Who Think Of Women As Ornaments Or Toys Only Good For Sex And Making Babies2. The feminist is unhappy. He or she asks for the phrase to be removed.

Let's say some person who is emphatically not a feminist notices this request for removal. Let's say the context is a society where men are generally portrayed in popular culture as violent bumbling apes who cause all world problems. A culture where women can go on for hours about what boors men are, but any man who says a word about women is immediately branded a sexist pig. A culture where a popular feminist once said that all sex was rape [EDIT: Or not. Apologies for misquote], and many people believed her, one with affirmative action laws mandating that women be hired over equally qualified men, one where you can't say "chairman of the board" without someone calling you sexist and accusing you of taking advantage of your male privilege to ignore male privilege if you disagree.

This request to remove a potentially offensive phrase is now a symbol of All Those Feminists Who Hate Men And Want Them To Feel Guilty All The Time For Vague Reasons. He or she gets angry, and certainly won't remove the offending phrase.

I'm not sure that's what's happening in this case, but I don't think a few poorly worded phrases followed by a polite request to change those poorly worded phrases would have reached five hundred fifty comments divided over four top-level posts if people were just taking it as a request to use slightly different language. In our completely hypothetical example, of course.

I call this mistake "missing the trees for the forest". If you have a specific case you need to judge, judge it separately on its own merits, not the merits of what agendas it promotes or how it fits with emotionally charged narratives3.

 

Footnotes

1: This gets worse once it gets formally organized into political parties. You get people saying something like "How can you, as an atheist, support the war in Iraq?" and thinking it makes perfect sense, because, after all, the war in Iraq is a Republican initiative, and the Republicans are the party of religious conservatives, therefore... Oh, yes, people think like this.

2: Oh, and this answers a question I sometimes hear asked half-seriously on message boards: how come derogatory jokes are okay in some settings but not in others? For example, how come Polish jokes are generally considered okay, but black jokes definitely aren't? Or how come it's considered okay for a black person to make a racist-sounding joke about black people or use the n-word, whereas it's not okay for a white person?

I think the answer is that if I were to make a Polish joke, it would be interpreted as what it is - a joke that needed somebody to play the part of a stupid person to be funny, and Polish people have traditionally served that role. There is no active well-known ongoing context of persecution of Polish people for the joke to symbolize, so it symbolizes nothing but itself and is inert. If I were to tell a joke about black people, even if it was clear that I wasn't actually racist and just thought the joke was funny, then since most people have a very active concept of persecution of black people, my joke would be a symbol of that persecution, and all right-thinking people who oppose that persecution would also probably oppose my joke. 

This leads to the odd conclusion that in a society known to be without racism, no one would mind racist jokes or slurs. In fact, this is confirmed by evidence. Black people are, society generally assumes, above suspicion when it comes to anti-black racism, and therefore black people can use the "n-word" without most people objecting.

This is what led to me developing some of these thoughts. I told a joke which I considered to be making fun of racism. Someone who heard it misinterpreted it and thought it was racist, accused me of racism, spread rumors that I was racist, and generally started a large and complicated campaign to discredit me. After that, I noticed that I was always coming to the defense of people who were accused of racism, and was willing to dismiss practically the entire concept of racism in society as a self-serving attempt at personal gain by minorities, a one hundred eighty degree turn from my previous attitude. Eventually I realized that I was just re-fighting the battle I had to fight after this one joke, and fitting everything to my "sometimes false accusations of racism unfairly harm majority group members and we need to protect against this" narrative. So I stopped. I think.

This also could explain why, contrary to Robin Hanson's hopes, people will never stop using disclaimers. They're ways of saying "I did this action for reasons that do not relate to your narrative; please exclude me from it", and this is not people's default position.

3: One objection could be that the specific case could start a slippery slope, or create a climate in which other things become viewed as more acceptable. In my experience, neither of these matter nearly as much as they would have to to justify the number of times people invoke them.

Comment author: Aurini 21 July 2009 03:42:49PM *  7 points [-]

TANSTAAFL: There Ain't No Such Thing As A Free Lunch

That's still a better justifcation of your behaviour than the MIT students [edit: Yvain did not actually say MIT!] used - not to mention that you're able to look back in retrospect and acknowledge the error of your decision.

This sort of suspicion is a good heuristic, if not the best heuristic. Scam artists (by which I mean casinos and carnies) are skilled at making things appear as if you've found the loophole in their game, and when you don't have enough time to examine the loophole thoroughly you're generally better off assuming it to be false. From the sound of things you were too busy to do this, not to mention that - being unfamiliar with multiple choice tests in general - it caught you with your pants down. You would have had to devote twice the analysis time as a typical North American, who is familiar with these sorts of exams.

Don't discount the TANSTAAFL heuristic - you made a rational choice based on limited data and available processing time. Your error is wholly different from the errors at MIT.

In response to comment by Aurini on Shut Up And Guess
Comment author: Yvain 21 July 2009 03:56:29PM *  24 points [-]

Ooh, this is interesting. Eliezer says he hopes this wasn't at MIT or somewhere, and now people are remembering the MIT reference and assuming I go to MIT. Reminds me of that bias where you try to debunk a rumor, and all people can remember is that they heard someone talking about the rumor somewhere and believe it more. What's that called? There was an OB article on it somewhere, I think.

I should hire Eliezer to come by and make offhanded MIT references during my job interviews.

In response to comment by Yvain on Shut Up And Guess
Comment author: ArthurB 21 July 2009 03:42:59PM *  2 points [-]

On this page, the cumulative refers to the probability of obtaining at most p successes. You want to run it with 30 and 9 which gives you the right answer, 2.14%

Or you could put in 30 and 20 which gives you the complement.

What is lower than 1% is the probability of getting 8 or less right answers.

In response to comment by ArthurB on Shut Up And Guess
Comment author: Yvain 21 July 2009 03:47:27PM 1 point [-]

Oh, I see how they did that. Thanks. Original post edited.

In response to Shut Up And Guess
Comment author: Eliezer_Yudkowsky 21 July 2009 05:03:58AM 17 points [-]

Oh my dear lord Cthulhu. Can I ask what level of class this was? If you say it was a postgraduate course at MIT, I may gather the last sane members of the human race and move to Pluto.

Comment author: Yvain 21 July 2009 03:35:27PM *  8 points [-]

Postgraduate course at a university that's not Ivy League caliber but reasonably well-respected. In contrast to ahem some of the comments below, these people are all quite smart, some consistently better able to understand difficult concepts than I and a few having good original published research. This sort of rationality stuff is just a different skill that some smart people just don't have aptitude in.

In response to Shut Up And Guess
Comment author: ArthurB 21 July 2009 02:37:25PM 6 points [-]

I think you got your math wrong

If you get 20 out of 30 questions wrong, you are break even, therefore the probability of losing points by guessing is

Sum( (i 30), i = 21..30) / 2^30 ~ 2.14% > 1%

In response to comment by ArthurB on Shut Up And Guess
Comment author: Yvain 21 July 2009 03:25:11PM *  0 points [-]

You're probably right, because I haven't done a problem like this since forever, but help me figure out what I did wrong. I found a binomial distribution calculator (this is binomial distribution, right?), entered 30 trials, 21 "successes", (counting a false answer as a success, and agreeing with you that 20 is break even so you need 21 to do worse than even) and .5 probability of success, and it said the cumulative probability was .9919... against, therefore <1%.

Shut Up And Guess

79 Post author: Yvain 21 July 2009 04:04AM

Related to: Extreme Rationality: It's Not That Great

A while back, I said provocatively that the rarefied sorts of rationality we study at Less Wrong hadn't helped me in my everyday life and probably hadn't helped you either. I got a lot of controversy but not a whole lot of good clear examples of getting some use out of rationality.

Today I can share one such example.

Consider a set of final examinations based around tests with the following characteristics:

* Each test has one hundred fifty true-or-false questions.
* The test is taken on a scan-tron which allows answers of "true", "false", and "don't know".
* Students get one point for each correct answer, zero points for each "don't know", and minus one half point for each incorrect answer.
* A score of >50% is "pass", >60% is "honors", >70% is "high honors".
* The questions are correspondingly difficult, so that even a very intelligent student is not expected to get much above 70. All students are expected to encounter at least a few dozen questions which they can answer only with very low confidence, or which they can't answer at all.

At what confidence level do you guess? At what confidence level do you answer "don't know"?

I took several of these tests last month, and the first thing I did was some quick mental calculations. If I have zero knowledge of a question, my expected gain from answering is 50% probability of earning one point and 50% probability of losing one half point. Therefore, my expected gain from answering a question is .5(1)-.5(.5)= +.25 points. Compare this to an expected gain of zero from not answering the question at all. Therefore, I ought to guess on every question, even if I have zero knowledge. If I have some inkling, well, that's even better.

You look disappointed. This isn't a very exciting application of arcane Less Wrong knowledge. Anyone with basic math skills should be able to calculate that out, right?

I attend a pretty good university, and I'm in a postgraduate class where most of us have at least a bachelor's degree in a hard science, and a few have master's degrees. And yet, talking to my classmates in the cafeteria after the first test was finished, I started to realize I was the only person in the class who hadn't answered "don't know" to any questions.

I have several friends in the class who had helped me with difficult problems earlier in the year, so I figured the least I could do for them was to point out that they could get several free points on the exam by guessing instead of putting "don't know". I got a chance to talk to a few people between tests, and I explained the argument to them using exactly the calculation I gave above. My memory's not perfect, but I think I tried it with about five friends.

Not one of them was convinced. I see that while I've been off studying and such, you've been talking about macros of absolute denial and such, and while I'm not sure I like the term, this almost felt like coming up against a macro of absolute denial.

I had people tell me there must be some flaw in my math. I had people tell me that math doesn't always map to the real world. I had people tell me that no, I didn't understand, they really didn't have any idea of the answer to that one question. I had people tell me they were so baffled by the test that they expected to consistently get significantly more than fifty percent of the (true or false!) questions they guessed on wrong. I had people tell me that although yes, in on the average they would do better, there was always the possibility that by chance alone they would get all thirty of the questions they guessed on wrong and end up at a huge disadvantage1.

I didn't change a single person's mind. The next test, my friends answered just as many "don't know"s as the last one.

This floored me, because it's not one of those problems about politics or religion where people have little incentive to act rationally. These tests were the main component of the yearly grade in a very high-pressure course. My friend who put down thirty "don't know"s could easily have increased his grade in the class 5% by listening to me, maybe even moved up a whole letter grade. Nope. Didn't happen. So here's my theory.

The basic mistake seems to be loss aversion2, the tendency to regret losses more than one values gains. This could be compounded by students' tendency to discuss answers after the test: I remember each time I heard that one of my guesses had been wrong and I'd lost points, it was a deep psychic blow. No doubt my classmates tended to remember the guesses they'd gotten wrong more than the ones they'd gotten right, leading to the otherwise inexplicable statement that they expect to get more than half of their guesses wrong. But this mistake should disappear once the correct math is explained. Why doesn't it?

In The Terrible...Truth About Morality, Roko gives a good example of the way our emotional and rational minds interact. A person starts with an emotion - in that case, a feeling of disgust about incest, and only later come up with some reason why that emotion is the objectively correct emotion to have and why their action of condemning the relationship is rationally justified.

My final exam, thanks to loss aversion, created an emotional inclination against guessing, which most of the students taking it followed. When confronted with an argument against it, my friends tried to come up with reasons why the course they took was logical - reasons which I found very unconvincing.

It's really this last part which was so perfect I couldn't resist posting about it. One of my close friends (let's call him Larry) finally admitted, after much pestering on my part, that guessing would increase his score. But, he said, he still wasn't going to guess, because he had a moral objection to doing so. Tests were supposed to measure how much we knew, not how lucky we were, and if he really didn't know the answer, he wanted that ignorance to be reflected in his final score.

A few years ago, I would have respected that strong committment to principle. Today, jaded as I am, I waited until the last day of exams, when our test was a slightly different format. Instead of being true-false, it was multiple-choice: choose one of eight. And there was no penalty for guessing; indeed, there wasn't even a "don't know" on the answer sheet, although you could still leave it blank if you really wanted.

"So," I asked Larry afterwards, "did you guess on any of the questions?"

"Yeah, there were quite a few I didn't know," he answered.

When I reminded him about his moral commitment, he said something about how this was different because there were more answers available so it wasn't really the same as guessing on a fifty-fifty question. At the risk of impugning my friend's subconscious motives, I think he no longer had to use moral ideals to rationalize away his fear of losing points, so he did the smart thing and guessed.

Footnotes

1: If I understand the math right, then if you guess on thirty questions using my test's scoring rule, the probability of ending up with a net penalty from guessing is less than one percent [EDIT: Actually just over two percent, thank you ArthurB]. If, after finishing all the questions of which they were "certain", a person felt confident that they were right over the cusp of a passing grade, assigned very high importance to passing, and assigned almost no importance to any increase in grade past the passing point, then it might be rational not to guess, to avoid the less than one percent chance of failure. In reality, no one could calculate their grade out this precisely.

2: Looking to see if anyone else had been thinking along the same lines3, I found a very interesting paper describing some work of Kahneman and Tversky on this issue, and proposing a scoring rule that takes loss aversion into account. Although I didn't go through all of the math, the most interesting number in there seems to be that on a true/false test that penalizes wrong answers at the same rate it rewards correct answers (unlike my test, which rewarded guessing), a person with the empirically determined level of human loss aversion will (if I understand the stats right) need to be ~79% sure before choosing to answer (as opposed to the utility maximizing level of >50%). This also linked me to prospect theory, which is interesting.

3: I'm surprised that test-preparation companies haven't picked up on this. Training people to understand calibration and loss aversion could be very helpful on standardized tests like the SATs. I've never taken a Kaplan or Princeton Review course, but those who have tell me this topic isn't covered. I'd be surprised if the people involved didn't know the science, so maybe they just don't know of a reliable way to teach such things?

Comment author: Yvain 04 July 2009 09:10:01PM 1 point [-]
Comment author: Yvain 02 July 2009 12:00:47PM 3 points [-]

One story is especially obnoxious, as Gawande recounts how a "gut feeling" about a case of infection causes him to persuade a patient to undergo an extensive biopsy which reveals she has life-threatening necrotizing fasciitis, caused by the popularly-known "flesh-eating bacteria". He explicitly acknowledges that he had no logical reason for regarding the case as unusual, that all appearances indicated a routine cellulitis which was approximately three thousand times more likely than necrotizing fasciitis was - and that he had recently taken care of a patient who had died agonizingly from a massive infection of the flesh-eating bacteria. And yet the story is presented as an example of how to deal with uncertainty.

I agree Gawande isn't especially virtuous here; it would be nice if he'd sought statistics on how often his hunches were correct. But don't condemn him too quickly either. There are some theories, even some ones with experimental support, that suggest intuitions are often the sum of reasoning processes that don't reach the conscious level. The case you describe: where everything seems okay but you have a nagging feeling in the back of your head that it isn't - is especially a red flag in that regard. I recommend Jonah Lehrer's How We Decide for more on this.

Comment author: [deleted] 02 July 2009 06:13:39AM 1 point [-]

Yes, but not wanting to think about apples also constitutes thinking about apples. However, if I do want to think about apples, I'm going to think about apples more than if I didn't want to think about apples. Perhaps a better example: if I think to myself, "I want to calculate 33 + 28", I will. Something's going on here; do you not agree?

In response to comment by [deleted] on Open Thread: July 2009
Comment author: Yvain 02 July 2009 11:53:15AM *  2 points [-]

Yes, but not wanting to think about apples also constitutes thinking about apples. However, if I do want to think about apples, I'm going to think about apples more than if I didn't want to think about apples.

That's not entirely true: look up Wegner, "Paradoxical Effects of Thought Suppression", 1987.

If my own experience is typical, people don't usually think "I want to think about apples" unless it's part of a thought experiment or something. A behaviorist model might work here: You get a stimulus: something that activates your brain's concept of apples. It may be a sense impression, like seeing an apple, or it may be a thought, for example a long train of thoughts about gravity and Isaac Newton eventually gets by a train of spreading activation to "apples". This stimulus gets processed by various different cognitive layers in various different ways that are interpreted by your conscious mind as "thinking about apples."

If you want to not think about apples for some odd reason, the natural tendency is for this to activate your apple concept and cause you to think about apples. If you're smart, though, you'll try to distract yourself by thinking about oranges or something, and since your conscious brain can only think about one thing at a time, this will probably work.

The breakthrough for me was realizing that "I think about apples" is more a peculiarity of the English language than a good reflection of what is happening - about as useful as "I choose to produce adrenaline in response to stress". It suggests that there's someone named me with a flashlight illuminating certain thoughts at certain times because I feel like it. I find it less wrong (though still a little wrong) to imagine the thoughts percolating up of their own accord, and me as a spectator. This might make more sense if you meditate.

Comment author: Eliezer_Yudkowsky 30 June 2009 12:39:36AM 5 points [-]

My name is Eliezer Yudkowsky... okay, that may not sound very promising, but I've been repeatedly told that my name "sounds just like a scientist's name".

Comment author: Yvain 30 June 2009 10:50:44AM 5 points [-]

I think that's more like the confounders they try to eliminate: Eliezer Yudkowsky is a very Eastern European / Jewish sounding name, and both Eastern Europeans and Jews are perceived as commonly being scientists.

The -sky ending also calls to mind Lobachevsky, Minsky, Korzybski, Tsiolkovsky, Tarski, and other really smart people.

In response to What's In A Name?
Comment author: JustinShovelain 29 June 2009 06:49:09PM 3 points [-]

Is this whole bias caused by the exposure effect? Would there be any obstacle in unifying the two? Do people also prefer to live in towns that are associated with their parents' names? Do people who fall for this effect also name their pets or children after themselves to a greater extent?

Comment author: Yvain 30 June 2009 10:37:40AM 1 point [-]

The second link in the article, the one with the words "they find", is a paper called "Name Letter Preferences Are Not Merely Mere Exposure". You should find some useful studies and stuff there.

If you want the paper but can't access it, tell me your email and I'll send it to you.

In response to What's In A Name?
Comment author: RichardKennaway 29 June 2009 02:46:28PM *  4 points [-]

That is all quite fascinating, in a "fancy that!" fashion, but whenever I see correlational data reported I wonder about the magnitude of the effect, and a measure of that magnitude in terms of bits of information. The first result they report is that if there were no influence between name and state of residence, the proportion of coincidences would be 0.1664, while the observed level is 0.1986. How large an influence does this represent?

I am not quite sure what the correct calculation to make is -- perhaps someone more versed in these matters can say -- but when I calculate the Kullback-Leibler divergence between two binary distributions, one with p=0.1664 and the other with p=0.1986, I get about 0.005 bits. When I estimate the mutual information between name and state, making various assumptions about the data I'd need for a precise calculation, I get a similar figure.

In short, if you want to predict someone's name from their state, or vice versa, the result is completely useless. Of course, making such a prediction was not the authors' purpose. But then, what was? What can you do with less than a hundredth of a bit?

How justifiable is it to report the finding in these words (quotes from the paper):

people are attracted to places that resemble their own names.

and

these findings challenge traditional assumptions about how people make major life decisions

I have just found where Andrew Gelman has blogged about this (search his blog for "Pelham"). I don't have time to read what he says at the moment, but his headlines indicate he doesn't rate it.

Comment author: Yvain 30 June 2009 10:33:55AM *  6 points [-]

Ah. I figured he must have done it at some point since the only copy of the PDF file I could find was on Andrew's site with the name "stuff-for-blog", but Google searches for "Gelman Pelham" and "Gelman name letter" didn't turn anything up. If I'd known I would have just linked him. I hope he's not upset that I'm "copying" him.

I did not first read this study on Gelman's blog. Actually, there is a story behind where I first read it. It was in a college psychology class. I was quite nervous throughout class that day, because I was going to ask the professor after class to write me a letter of recommendation for a postgrad program in Scotland I wanted to get into. We spent an hour or so going over this paper and implicit egoism, and then after class I asked the professor to help me get into the program, and she started cracking up.

...see, my real name is Scott, and it was a program in Scotland, and we'd just finished studying the name letter effect...the next day she told the entire class about it, and I was suitably embarrassed, and the name letter effect has stuck in my memory ever since.

In response to What's In A Name?
Comment author: Alicorn 29 June 2009 03:50:02PM 7 points [-]

A lot of relatively weighty decisions wind up being made for trivial reasons simply because all of the non-trivial factors cancel each other out - for instance, if I were trying to decide whether to go into ethics or metaphysics (a choice with long-term career impact, assuming I get to be a professor one day) and I didn't find myself strongly preferring one over the other, I could see myself picking one for a silly reason. If my name were Ethel, which it is not, and I liked the sound of "Ethel the ethicist", that might tip the balance. Either that, or contemplating that choice would throw into sharp relief something I'd been overlooking in favor of metaphysics. But if there is no such factor, then why not choose on the basis of "Ethel the ethicist" sounding nice? It arguably makes slightly more sense than flipping a coin.

In response to comment by Alicorn on What's In A Name?
Comment author: Yvain 30 June 2009 10:25:02AM 2 points [-]

If you haven't already, read the part of the paper where they talk about hardware and roofing stores. They ran some clever analyses to see whether the effect was caused by a love of alliteration (for example someone named Herman decides to go into hardware so he can call his store Herman's Hardware) and the results suggested this wasn't the explanation.

What's In A Name?

41 Yvain 29 June 2009 12:54PM

   Marge: You changed your name without consulting me?
   Homer: That's the way Max Power is, Marge.  Decisive.
      --
The Simpsons

In honor of Will Powers and his theories about self-control, today I would like to talk about my favorite bias ever, the name letter effect. The name letter effect doesn't cause global existential risk or stock market crashes, and it's pretty far down on the list of things to compensate for. But it's a good example of just how insidious biases can be and of the egoism that permeates every level of the mind.

The name letter effect is your subconscious preference for things that sound like your own name. This might be expected to mostly apply to small choices like product brand names, but it's been observed in choices of spouse, city of residence, and even career. Some evidence comes from Pelham et al's Why Susie Sells Seashells By The Seashore:

The paper's first few studies investigate the relationship between a person's name and where they live. People named Phil were found more frequently than usual in Philadelphia, people named Jack in Jacksonville, people named George in Georgia, and so on with p < .001. To eliminate the possibility of the familiarity effect causing parents to subconsciously name their children after their place of residence, further studies were done with surnames and with people who moved later in life, both with the same results. The results held across US and Canadian city names as well as US state names, and were significant both for first name and surname.

In case that wasn't implausible enough, the researchers also looked at association between birth date and city of residence: that is, were people born on 2/02 more likely to live in the town of Two Harbors, and 3/03 babies more likely to live in Three Forks? With p = .003, yes, they are.

The researchers then moved on to career choices. They combed the records of the American Dental Association and the American Bar association looking for people named either Dennis, Denice, Dena, Denver, et cetera, or Lawrence, Larry, Laura, Lauren, et cetera. That is: were there more dentists named Dennis and lawyers named Lawrence than vice versa? Of the various statistical analyses they performed, most said yes, some at < .001 level. Other studies determined that there was a suspicious surplus of geologists named Geoffrey, and that hardware store owners were more likely to have names starting with 'H' compared to roofing store owners, who were more likely to have names starting with 'R'.

Some other miscellaneous findings: people are more likely to donate to Presidential candidates whose names begin with the same letter as their own, people are more likely to marry spouses whose names begin with the same letter as their own, that women are more likely to show name preference effects than men (but why?), and that batters with names beginning in 'K' are more likely than others to strike out (strikeouts being symbolized by a 'K' on the records).

If you have any doubts about the validity of the research, I urge you to read the linked paper. It's a great example of researchers who go above and beyond the call of duty to eliminate as many confounders as possible.

The name letter effect is a great addition to any list of psychological curiosities, but it does have some more solid applications. I often use it as my first example when I'm introducing the idea of subconscious biases to people, because it's clear, surprising, and has major real-world effects. It also tends to shut up people who don't believe there are subconscious influences on decision-making, and who are always willing to find some excuse for why a supposed "bias" could actually be an example of legitimate decision-making.

And it introduces the concept of implicit egoism, the tendency to prefer something just because it's associated with you. It's one possible explanation for the endowment effect, and if it applies to my beliefs as strongly as to my personal details or my property, it's yet another mechanism by which opinions become calcified.

This is also an interesting window onto the complex and important world of self-esteem. Jones, Pelham et al suggest that the name preference effect is either involved in or a byproduct of some sort of self-esteem regulatory system. They find that name preferences are most common among high self-esteem people who have just experienced threats to their self-esteem, almost as if it is a reactive way of saying "No, you really are that great." I think an examination of how different biases interact with self-esteem would be a profitable direction for future research.

Comment author: pjeby 27 June 2009 10:31:09PM 3 points [-]

There are lots of PCT textbooks out there; I wrote based on two of them (combined with my own prior knowledge): "Behavior: The Control Of Perception" by William T. Powers, and "Freedom From Stress", by Edward E. Ford. The first book has math and citations by the bucketload, the latter is a layperson's guide to practical PCT applications written by a psychologist.

Comment author: Yvain 28 June 2009 07:37:25PM 12 points [-]

Wait a second. There's a guy who writes textbooks about akrasia named Will Powers? That's great.

Comment author: Yvain 23 June 2009 03:03:45PM *  6 points [-]

I don't think this post adequately distinguishes between two concepts: how does the human utility function actually work, and how should it work.

The answer to the first question is (I thought people here agreed) that humans weren't actually utility maximizers; this makes things like your descriptive argument against perceptive determinism unnecessary and a lot of your wording misleading.

The second question is: if we're making some artificial utility function for an AI or just to prove a philosophical point, how should that work - and I think your answer is spot on. I would hope that people don't really disagree with you here and are just getting bogged down by confusion about real brains and some map-territory distinctions and importing epistemology where it's not really necessary.

Comment author: Hey 18 June 2009 10:20:34AM 7 points [-]

I have been using this exact method for a few years. It is absolutely the most reliable method for getting something specific and critical done in an intermediate time frame (say 2 weeks to 3 months), but it's kind of the nuclear option of willpower and should be used sparingly since 1) it relies on being the nuclear option, if you ever fail then you would lose faith in the method 2) it absolutely sucks, since it's usually something sucky you decide to do and you have bargained away the usual weaseling out tactics 3) Cthulhu doesn't like it when you break your promises.

In response to comment by Hey on Applied Picoeconomics
Comment author: Yvain 19 June 2009 01:27:14PM 10 points [-]

it absolutely sucks, since it's usually something sucky you decide to do and you have bargained away the usual weaseling out tactics

That's a really good point. Robin likes to talk about this. Someone may enjoy eating fatty foods more than they would enjoy being fit and healthy. But people who express a desire to be fit and healthy get more social prestige, so the optimum case for them is to think they would be better off dieting, while continuing to eat as much as always. These people think they have akrasia, but don't. If someone gives them a way to "cure" their "akrasia", they'll just end out unhappy.

I got the impression that Robin thinks this explains most or all akrasia; I wouldn't go that far, but I think it explains some.

Comment author: Z_M_Davis 18 June 2009 06:22:55PM 9 points [-]

I don't actually have any numbers on hand, and to be clear, I don't claim to have achieved any level of sheer awesomeness, but rather only that I'm a better person than I used to be. (This is by no means a high bar.) You ask, how long have I been doing this---but I can't point to any discrete start; my personality has been in a sort of gradual flux in what I've been calling "these days of Eliezer Yudkowsky and my purity born of pain"---dating back to my nervous breakdown of 29 November 2007.

This was actually sort of my point: life is continuous. When you have a discrete goal, an explicit program with a start date and an end date, you can just fail. Whereas when you have an open-ended concept of things-worth-doing, there's no failure, only degrees of win. You seemed to be saying that when you have an open-ended goal, that just gives you an excuse to cheat. Whereas I'm working under the theory that if I want to cheat, I've already lost.

All this might tie into why I can't deal with school: they give you a curriculum, and all the good thoughts you have that aren't on the curriculum don't count, and everything that is on the curriculum that you didn't do is a mark of sin upon your soul, because you have a duty to perfectly obey the teacher's commands. It's too precise---arbitrarily precise. I realize that most people probably aren't like this---somehow they can muddle through the system without being driven to madness by all the little details. Most people have not thrown crying fits contemplating how they got a B in that poetry class, even though they weren't sure they did all of the reading, and therefore might not have truly deserved that B. Nor is it common, I imagine, to worry about what constitutes a "reading"---they tell you to read the chapter, but what does that mean?---if my eyes skim over a paragraph, do I have to go back and make sure I touch every word? I just want to be good! What do you want from me?!

I tried to obey. I couldn't. What would constitute obedience is either underspecified or overspecified; I can't figure out which. I need a different methodology if I am simply going to exist. Explicit rules and goals sum over far too many details. It might help to have handy verbal guidelines and best practices, but ultimately these are only useful if you really care about what you're doing. And if you really care about what you're doing, then you can't be so utterly dependent on the guidelines. "How will you discover your mistake? Not by comparing your description to itself, but by comparing it to that which you did not name." I really think it is a tautology that you should always be doing the best thing you could possibly be doing. You can't stop time; you're always going to be doing something, even if that something is nothing in particular. And so if you have to do some uniquely determined thing---it should be the best thing. But I think I repeat myself.

Comment author: Yvain 19 June 2009 01:17:16PM *  7 points [-]

Did you go to school, did you go to school for a while and then leave, or are you entirely self-taught?

Your method is clearly better if you are able to think like that successfully, and my method is mostly born from the observation that I can't. I've heard it said that one of the effects of spending a decade or two in the school system is that it twists your mind to think more in the way typical of my system and less in the way typical of yours. And I find that people who managed to avoid school almost entirely, like Eliezer, radiate a sort of psychological healthiness I can only dream of.

I had the same feelings about school as you did, my parents refused to let me leave, and I ended out, over a few years, becoming the sort of person who could tolerate the school experience. Sometimes I worry that the process made me less able to do a lot of other things, like strive for excellence in the way you're describing.

Comment author: MichaelVassar 18 June 2009 04:10:33PM 2 points [-]

The sample size issue etc is why I talk about Bayes. You get important info from single data points all the time in life. There's just a fetish against doing so in science due to bad epistemology trying and failing to counter other bad epistemology.

You certainly derived your belief that your procedure would work from a theory. You hadn't actually even seed it work, so nothing but a theoretical basis could explain your attempt.

Comment author: Yvain 18 June 2009 04:45:46PM 7 points [-]

I don't think it's second-order good epistemology trying and succeeding to counter bad epistemology.

Let's say we run a study with 30 people, and we conclude ZM's method is the best, with p = .55 (sorry, I don't think in Bayesian when I have my psychology experimentation cap on), which is realistic for that kind of sample and the variability we can expect. Now what?

We could come up with some kind of hokey prior, like that there's a 33% chance each of our techniques is best, then apply that and end up with maybe a 38% chance ZM's is best and a 31% chance mine and Pjeby's are best (no, I didn't actually do the math there). But first of all, that prior is hokey. Pjeby's a professional anti-procrastination expert, and we're giving him the same prior as me and Z.M. Davis? Second of all, we still don't really know what "best" means, and it's entirely possible different methods are best for different people in complex ways. Third, I don't trust anyone including myself to know what to do with a 7% chance. I like my method better; should I give that up just because a very small study ended up shifting the probabilities 7% toward ZM? Fourth of all, we still wouldn't know how to apply this to picoeconomics as a theory: using any technique will increase success by placebo effect alone, we have several techniques that all use picoeconomics to different degrees, and we would have to handwave new numbers into existence to calculate things and probably end up with something like a .1% or .2% shift in probabilities.

And this is all if we have perfect study design, there's no confounders, so on and so forth. It would take a lot of work. The best case scenario is that all that work would be for a single digit probability shift, and the realistic case is that there's flaw somewhere in the process, or we simply misinterpret the result (my guess is that people can't deal with a 2% shift correctly and just think "now there's evidence" and count the theory as a little more confirmed) then we'll actually be giving ourselves negative knowledge.

I'm not saying Bayes isn't useful, but it's useful when we have a lot of numbers, when we're willing to put in a very large amount of work, and where there's something clear and mathematical we can do with the output.

Comment author: pjeby 17 June 2009 05:47:37PM *  11 points [-]

My brain clearly has different procedures for dealing with vague responsibilities it can weasel out of, and serious responsibilities it can't, and the oath served to stick studying on the "serious" side of the line.

I doubt it's the oath or the rituals. The key piece (in my experience) that makes this work, is the part where you considered conflicts and consequences. You made explicit under which specific conditions you would do it, and which ones you wouldn't.

To translate this back into Ainslie's model, the success of bargaining at any point in time is dependent on the degree of activation of "interests". If at the time you decide to do something, you envision only the default case, then the interest is only mentally linked to the default, not the situation where "something else comes up".

However, if you explicitly contemplate all the things that might come up, and decide what you'll do in each case, then you are mentally linking your "interest" to those contexts, along with a preferred behavior... thus reducing the willpower load required to make those decisions when the time comes, and giving that "interest" a larger say in the bargaining that occurs at that point in time.

I mentioned this briefly in my New Years' Resolutions video this year, but I've been teaching this concept for some time now. A key variable is how well you can mentally put yourself into the conflict situations that will occur, so that you can actually make realistic tradeoffs in "near" thinking, rather than using over-idealistic "far" thinking.

Related is Martha Beck's concept of "four day wins", which emphasizes the idea that as long as you make changes to behavior in small enough steps that no unmanageable conflicts arise, you can make that behavior "normal" within four days.

For example, the 4-day win approach to studying 2 hours a day would first be to set a time at which you will simply take out your books and look at them for say, 1 minute, without actually opening them, and to do that for four days. Then, for four days, do that and actually open them to what you're supposed to be studying, but don't do any of it. From there on, setting timers for small intervals where you have to stop as soon as the timer goes off.

In either case, whether you do it by exhaustive imagination in advance (as you did), or the 4-day wins method, you are doing the same thing: establishing a conditioned link between the "interest" (in Ainslie's terminology) and the context where you would like that interest to be active, so as to prevent other interests from dominating the negotiation in that context.

My suggestion for your next experiment would be for you to see how much less ritual you can get away with, as long as you satisfy the criterion of linking your preferred decision and the feeling of your "interest" to as many explicit, specific contexts where other interests might be in conflict.

with the knowledge that if I succeeded I would have a great method of self-improvement and if I failed I would be dooming myself to a life of laziness forever (Ainslie's theories suggest that exaggeration is good in this case).

So do Tony Robbins's theories, as described in his 1991 book, "Awaken the Giant Within". However, this is an area where I disagree with both Robbins and Ainslie!

For certain personality types, creating this sort of bargain is dangerous and damaging if you fail, for reasons that are actually brought up in Ainslie's book. And those personality types are very likely to fail, for the simple reason that they will not actually do the critical task of considering all the ways in which they might fail.

For example, how many Republican senators and Baptist ministers would you guess have sworn mighty oaths to never have gay sex?

If the subject of your oath is an ideal-belief-reality conflict, you will not "negotiate fairly" with your other interests, which means you won't really think about the areas where conflicts will arise, which means those other interests will dominate, and your oaths will eventually fall by the wayside.

So I'm glad it worked for you, but I strongly recommend that others NOT add this sort of "leverage" - it's not necessary and can be damaging. The critical factor is inter-interest linkage, not consequences of failure.

Richard Bandler actually encourages people embarking on something new that they might backslide on, to actually visualize themselves backsliding... repeatedly! And then to imagine this making them MORE motivated to proceed.

And if you think about it, this is actually an example of the same principle I described above - linking an interest to a context where a conflict might occur. In this way, even if backsliding does occur, it still contributes to the goal, rather than detracting from it.

In contrast, what you have done sets up an expectation that a single failure will lead to the destruction of someone's entire life... and that is NOT a responsible thing to suggest or prime, EVER.

Comment author: Yvain 18 June 2009 04:12:52PM 5 points [-]

I'm going to have to read this a few more times before I understand it fully, but I want to address one thing right away:

For example, how many Republican senators and Baptist ministers would you guess have sworn mighty oaths to never have gay sex...what you have done sets up an expectation that a single failure will lead to the destruction of someone's entire life... and that is NOT a responsible thing to suggest or prime, EVER.

The way I dealt with this was to make my oaths in one month blocks. So the Republican would have to swear "I won't have any gay sex...this month." Even for the most lustful, this should be possible.

If, at the end of the month, this was so painful he wants to just give up on this, he can. Or if he thinks he can do it, he could also include the statement in his next month's oath.

What I found was that there's a very different mental feeling between "I can never do this again" and "I have to wait a month to do this." The latter is annoying but bearable, and it's why I included the "never make an open-ended oath" point up there.

(if you want to test this for yourself, but don't have any repressed homosexual urges, masturbation makes a good test case)

I don't think this technique is at its best for something where doing it once is a disaster, like gay sex for Baptist ministers. I think it's better for something like dieting. Tell yourself you won't eat cookies the whole month, do it in the full knowledge that you'll start eating cookies again when the oath runs out, pig out on cookies for one day, and then when you have no desire whatsoever for any more cookies, swear to diet again for the next month.

Comment author: Z_M_Davis 17 June 2009 05:47:55PM 34 points [-]

(I trust I will be forgiven for the overwrought and repetitive prose that follows. In my defense, on this issue, I really do try to think in such terms, and arguably all this drama is a large part of why the method works as well as it does.)

My improvement program, which has been working fairly well so far, although I am still continually refining things as I will detail below, is based on the opposite principle. Rather than setting explicit measurable goals, I try to continually remind myself that every minute and every dime is precious, and every minute and every dime that you don't spend doing the best thing you can possibly be doing is a mark of sin upon your soul, and furthermore that this is not some extremist dictate, but rather a tautology---that's what the word "best" means: that which you should be doing. Rather than goals to satisfice, I want to have a utility function to maximize. I do not place myself under some dreaded burden to fulfill some oath: I'm just trying to not be stupid. There is no such thing as "leisure"---everything is booked under "Dayjob" or "Lifework" or "Education" or "Maintenance," for every book that you read makes you stronger, every problem that you solve increases your beauty, every line that you write is another stitch in your ball gown. It is not: "Once I finish my homework, I can watch the teevee or play flash games on the internet." As an autodidactic generalist, I either have no homework, or an infinite amount of homework, depending on how you want to phrase things. I don't want to watch the goddam teevee! Mathematics is more fun than those moronic flash games! Slacking off is not a guilty indulgence; it's just stupid, and the entirety of my powers are now devoted to the monumental task of not-being-stupid. I recognize no other intertemporal selves to bargain with---I have but one Self, a timeless abstract optimization process to which this ape is but a horribly disfigured approximation. There have been times when I was tempted to go buy an ice cream ("frozen yogurt") and even took a few steps towards the shop before thinking---is this really what I want? Living as I am on short time, wouldn't have rather have that four dollars which is equivalent to twenty-four minutes at my crappy dayjob? I prefer the money, so I turned and walked back to my car.

All this is not to say I am in no need of more structure---it would be helpful to keep some sort of schedule or timelog, not in the form of an oath to another self from another time, but simply as a guideline to give direction to my full autodidactic fury. I've experimented with this and that, to no notable success so far---but I'm going to keep hacking away at this; sunk costs can't play into your decision theory, so no number of failures can discourage an expected utility maximizer, though such a thing might happen to a goddam ape.

Am I kidding myself?---in some sense, maybe a little. How much writing have I done?--when allegedly my lifework was supposed to be a work of fiction. Does it only seem like I've been being more efficient, because I've been doing so much math and programming which leaves a paper trail, as compared to reading which doesn't? But for once in my life, induction is on my side now: I've gotten better before, so I can do so again. I don't watch teevee any more, and I don't play flash games---I'm not even tempted. I don't know what my limits are. So help me.

Comment author: Yvain 18 June 2009 04:05:44PM 14 points [-]

Can you give us some numbers? How long have you been doing this? What is your average day (better yet: yesterday) like?

I tried something like this when I was very young - middle school, maybe. I think the most embarrassing part was where I decided I would never have any interest in the opposite sex, because that would be a distraction. It lasted for about a week before I decided maybe there was something to this "being human" thing after all, and put it all down to childhood exuberance and never tried anything of that sort again.

...but if you can actually pull it off, you are my new hero.

Comment author: Yvain 18 June 2009 03:42:37PM *  3 points [-]

Really, really, really doubtful that correlations between national IQ and, well, anything prove anything besides that certain countries are generally better off than others. That correlation is probably just differentiating First World countries from Third World countries in general - the First World has better health and education, and also better government. Although I'm agnostic on the existence of racial IQ differences, those aren't what's going on here, considering the wide variation in success of countries with similar races.

Same with IQ versus religion within and between countries: it's probably just an artifact of religion vs. wealth correlations. I scanned those articles and I didn't see anything saying they'd adjusted for it; if there is, then I'll start getting excited.

Comment author: Yvain 18 June 2009 03:24:10PM 5 points [-]

My first response to this was "Wait, I have a model?"

Right now I don't think we're even at a point where it's valuable to think of three different "theories" or what they imply. We have three different techniques, all of which are kind of supported by picoeconomics but also kind of hand-wave-y. And ZM's objection to my post seems to be more philosophical than a simple "your method won't work" (and I still don't entirely understand Pjeby).

Once we have some more discussion, if it becomes clear that we have actually have three different but comparable willpower techniques and we really want to know which will work best, then we can start wishing we could test them. I'm doubtful we can actually do so, since I doubt we could get more than about 30 volunteers, and 30 divided by 3 groups does not a legitimate sample size make for a complicated psych experiment. But if you have some ideas, I'd be happy to help.

Applied Picoeconomics

46 Post author: Yvain 17 June 2009 04:08PM

Related to: Akrasia, Hyperbolic Discounting, and Picoeconomics,  Fix It And Tell Us What You Did

A while back, ciphergoth posted an article on "picoeconomics", the theory that akrasia could be partially modeled as bargaining between present and future selves. I think the model is incomplete, because it doesn't explain how the analogy is instantiated in the real world, and I'd like to investigate that further sometime1 - but it's a good first-order approximation.

For those of you too lazy to read the article (come on! It has pictures of naked people! Well, one naked person. Suspended from a graph of a hyperbolic curve) Ainslie argues that "intertemporal bargaining" is one way to overcome preference reversal. For example, an alcoholic has two conflicting preferences: right now, he would rather drink than not drink, but next year he would rather be the sort of person who never drinks than remain an alcoholic. But because his brain uses hyperbolic discounting, a process that pays more attention to his current utility than his future utility, he's going to hit the whiskey.

This sticks him in a sorites paradox. Honestly, it's not going to make much of a difference if he has one more drink, so why not hit the whiskey? Ainslie's answer is that he should set a hard-and-fast rule: "I will never drink alcohol". Following this rule will cure his alcoholism and help him achieve his dreams. He now has a very high preference for following the rule; a preference hopefully stronger than his current preference for whiskey.

Ainslie's other point is that this rule needs to really be hard-and-fast. If his rule is "I will drink less whiskey", then that leaves it open for him to say "Well, I'll drink some whiskey now, and none later; that counts as 'less'", and then the whole problem comes back just as bad as before. Likewise, if he says "It's my birthday, I'll let myself break the rule just this once," then soon he's likely to be saying "It's the Sunday before Cinco de Mayo, this calls for a celebration!" Ainslie has some much more formal and convincing ways of framing this, which is why you should read the article instead of just trusting this summary.

The stuff by Ainslie I read (I didn't spring for any of his dead-tree books) didn't offer any specific pointers for increasing your willpower2, but it's pretty easy to read between the lines and figure out what applied picoeconomics ought to look like. In the interest of testing a scientific theory, not to mention the ongoing effort to take control of my own life, I've been testing picoeconomic techniques for the last two months.

The essence of picoeconomics is formally binding yourself to a rule with as few loopholes as possible. So the technique I decided to test3 was to write out an oath detailing exactly what I wanted to do, list in nauseating detail all of the conditions under which I could or could not be released from this oath, and then bind myself to it, with the knowledge that if I succeeded I would have a great method of self-improvement and if I failed I would be dooming myself to a life of laziness forever (Ainslie's theories suggest that exaggeration is good in this case).

I chose a few areas of my life that I wanted to improve, of which the only one I want to mention in public is my poor study habits. I decided that I wanted to increase my current study load from practically never looking at a book after school got out, up to two hours a day.

I wrote down - yes, literally wrote down - an oath in which I swore to study for two hours a day. I detailed exactly the conditions that would count as "studying" - no watching TV with an open book placed in my lap, for example.

I also included several release valves. The theory behind this was that if I simply broke the oath outright, the oath would no longer be credible and would lose its power (again, see Ainslie's article), and there would be some point where I would be absolutely compelled to break the oath (for example, if a member of my family is in the emergency room, I refuse to read a book for an hour and a half before going to check up on them). I gave myself a whole bunch of cases in which I would be allowed to not study, guilt-free, and allowed myself five days a month when I could just take off studying for no reason (too tired, maybe). I also limited the original oath to a month, so that if it didn't work I could adjust it without completely destroying the effectiveness of the oath forever. Finally, I swore the oath in a ceremonial fashion, calling upon various fictional deities for whom I have great respect.

One month later, I find that I kept to the terms of the oath exactly, which is no small achievement for me since my previous resolutions to study more have ended in apathy and failure. On an introspection level, the need to study each day felt exactly like the need to complete a project with a deadline, or to show up for work when the boss was expecting you. My brain clearly has different procedures for dealing with vague responsibilities it can weasel out of, and serious responsibilities it can't, and the oath served to stick studying on the "serious" side of the line.

I am suitably cautious about other-optimizing and the typical mind fallacy, so I don't promise the same method will work for you. But I'd be interested to see if it did4. I'd be especially interested if everyone who tried it would post, right now, what they're trying so that in a month or so we can come back and see how many people kept their oath without having too much response bias.

 

Footnotes

1: I'm split on the value of picoeconomic theory. A lot of it seems either common-sense if taken as a vague model or metaphor, or obviously false if taken literally. But sometimes it's very good to have a formal model for common sense, and I'm optimistic about someone developing a more literal version of it that explains what's actually going on inside someone's head.

2: Ciphergoth, as far as you know does Ainslie ever start making practical suggestions based on his theory anywhere, or does he leave it entirely as an exercise for the reader?

3: I don't read a lot of stuff on productivity, so I might be reinventing the wheel here.

4: For people trying this, a few suggestions and caveats from my experience:

  1. Do NOT make the oath open-ended. Set a time limit, and if you're happy at the end of that time limit, set another time limit.
  2. Don't overdo it; this only works if you really do want the goal you're after more than you want momentary pleasure, people are notoriously bad at knowing what they want, and if you break an oath once you've set a precedent and it'll be harder to keep a better-crafted oath next time. If I'd sworn six hours of studying a day, no way I'd have been able to keep it.
  3. Set release valves.
  4. Do something extremely measurable in which success or failure is a very yes-or-no affair, like how much time you do something for. Saying "study more" or "eat better" will be completely useless.
  5. Read the article so you know the theory behind it and especially why it's important to always keep the rules.
  6. Don't just think up the oath and figure it's in effect. Write it down and swear it aloud, more or less ceremonially, depending on your taste for drama and ritual.
  7. Seriously, don't overdo it. Ego depletion and all that.
Comment author: Yvain 17 June 2009 03:55:04PM *  1 point [-]

The one sentence summary is a very surprising result. The article made a little more sense when it talked about announcing "satisfactions" (like "I've just bought running shoes, so my plan to run more is going well") versus "dissatisfactions" (like "I really need to run more now that I've got these shoes").

I predict that there will be a strong interaction with type of goal. If it's a goal that can be fudged, like "I'm going to eat less", then probably telling your friends you're going to eat less will cause you not to do it because you've already gotten the status boost and no one can really call you on whether you've succeeded. If it's a goal where completion is obvious, like "I'm going to go back to college", then telling your friends would be a public committment that puts you under pressure to follow through.

Possibly related: Trying to Try

Comment author: davidr 15 June 2009 03:47:48PM 1 point [-]

"Everything that happens happens as it should, and if you observe carefully, you will find this to be so. " - Marcus Aurelius

Comment author: Yvain 15 June 2009 03:50:39PM *  5 points [-]

I counter with:

Many things do not happen as they should, and most things do not happen at all. It is the duty of the conscientious historian to correct these defects.

-- Herodotus

Comment author: Annoyance 15 June 2009 02:39:31PM -2 points [-]

I will read your next response, but otherwise not continue this thread further.

I'm not interested in conversing with people who make long lists of assertions, then remove themselves from the discussion.

Comment author: Yvain 15 June 2009 03:41:14PM *  3 points [-]

...sigh. Okay, put it like this. We're clearly arguing past each other. I think your points are self-evidently wrong, and your arguments bordering on trolling. I am sure this is not how the discussion appears to you, and you may feel that my points are equally bad, but we're not making any progress here. And it's degenerating into a Standard Political Debate - basically a libertarian "no coercive government is ever okay" position versus a utilitarian "sometimes it's an optimal solution" position, which has been done about a billion times and about which there is very little left to be said.

That leaves us with two options. We can either continue unproductively wasting time and energy on a particularly unproductive version of a cliched topic that neither of us can realistically affect, all the while breaking the Less Wrong gentlemens' agreement against explicit political discussions. Or one person can bow out and allow the other person to take the last word.

Comment author: Yvain 15 June 2009 09:57:40AM 26 points [-]

"Voting in a democracy makes you feel powerful, much as playing the lottery makes you feel rich." -- Mencius Moldbug

Comment author: Yvain 15 June 2009 09:53:01AM *  4 points [-]

"Train yourself to get suspicious every time you see simplicity. Any claim that the root of a problem is simple should be treated the same as a claim that the root of a problem is Bigfoot. Simplicity and Bigfoot are found in the real world with about the same frequency." -- David Wong

In response to Readiness Heuristics
Comment author: Yvain 15 June 2009 09:36:50AM *  8 points [-]

Annoyance didn't mention this explicitly in his post about frontal syndrome, but these readiness heuristics are related to the frontal lobe of the brain in some way, and can be damaged with interesting results. Damasio describes a patient with a frontal stroke who lost the ability to terminate the decision-making process. He told the story of (I think I remember this right) trying to schedule an appointment with this patient, and told him that either Tuesday or Thursday would work, and the patient spent the next several minutes listing all the possible reasons why one day might be better than the other and balancing them out. When it became overwhelming, Damasio finally just said "What about Thursday?" and the patient immediately agreed.

Comment author: Yvain 15 June 2009 09:24:40AM *  13 points [-]

"Imagine a world where everything changes to match the state of your mind, where evidence never pushes back against your theories, where your every thought is correct simply because you think it so. Can there be any better definition of hell for a man of learning? "

-- Bradeline, Fall From Heaven

Comment author: Annoyance 12 June 2009 05:28:22PM *  2 points [-]

a fully general counterexample to all possible policies

Not to all possible policies, and not fully general. Frankly, I'm rather surprised that you don't see any alternative solutions.

It is perfectly possible to reward 'courtesy', and punish 'discourtesy', without resorting to government regulation - or regulation of any kind.

If you have six roommates, one of whom is inconsiderate, you have a number of options. You can be inconsiderate towards the rude person in an attempt to dissuade their inconsideration. You can try to persuade the other five to exert social pressure on your behalf. You can move out.

Sometimes you can't get what you want, period. The people who don't acknowledge that point either never consider that they could be on the receiving end of power, or believe that they have enough power and dominance to ensure that it'll never happen to them.

What if your roommates - and everyone who learns of your conflict - believes you're the one being inconsiderate, and the 'inconsiderate' person to be behaving quite appropriately? Would you be willing to be subject to the same coercion you favor for others?

Comment author: Yvain 12 June 2009 05:59:50PM *  8 points [-]

You seem to be saying that once we accept the possibility of ever coercing anyone, we have to also accept the possibility of coercion being misused. You then suggest that since we don't want coercion to be misused, we can never coerce anyone, and we should accept a society where other people can do whatever they want.

This is a lot like saying that since science could theoretically be used to bioengineer a plague, we should avoid all scientific thought.

I don't demand - as you seem to think I do - that everyone do whatever I want. I demand that everyone work together for a solution that maximizes the utility of everyone. I believe that a society where we all realize that no one raping anyone leaves everyone better off is better than a society where everyone can rape whoever they want. Likewise, I think a society with certain minimum noise restrictions will leave everyone, whether noisy or quiet, in general better off than one where everyone is free to extort their neighbors for however much they want.

This isn't new - Bentham and Mill worked out the details several hundred years ago. Yes, there are costs from the existence of enforcement mechanisms and the potential for the restriction of freedom to be greater than the benefits. But in some cases - like the case of please don't rape people - the benefits are clearly greater than the costs.

Sometimes you can't get what you want. But most people who enjoy proclaiming that very loudly are just trying to signal how hard-headed and tough they are. If there's an easy way in which you can get what you want, there's no extra virtue in refusing to take it. Having restrictions about not committing violence against other people is one such easy way.

I am not trying to say that I've thought about it and I'm absolutely sure there's no possible non-coercive way to solve the problem of rape. If you can think of one, you're welcome to post it. I'm just trying to say that your particular argument here that all coercive methods are bad doesn't hold any water.

[addendum: no, I don't think the case of violence and annoyance are particularly different. If it helps, imagine a person releasing poison gas from the room next door. If the gas kills me, it's violence. If it's a little less gas, and it merely injures me to such a degree I end up in the hospital for a month, it's still violence. If the gas sends me into a fit of coughing every time I breathe, it's annoyance. If it just makes me itchy, it's definitely annoyance. At what point does releasing the gas change from "injury" to "annoyance"? I would say these are artificial categories with no real-world equivalent, and that instead of looking for a clean answer with an obvious distinguishing case, you have to just accept that there's going to be a cost-benefit analysis to going over to your neighbor's and smashing the gas-apparatus either way, and that at some points it will return negative and at other points positive results.]

This is turning into a political discussion here, and not even one that meets this community's high standards. I will read your next response, but otherwise not continue this thread further.

Comment author: RichardKennaway 12 June 2009 02:09:01PM 13 points [-]

If people's political opinions come partly from unchangeable anatomy

It's a big step to go from finding physical correlates of mental phenomena -- especially hypothetical ones -- to ascribing the mental phenomena to "unchangeable anatomy".

Comment author: Yvain 12 June 2009 05:20:45PM 4 points [-]

I accept your correction.

Comment author: Annoyance 12 June 2009 05:04:39PM 4 points [-]

Perhaps you'd prefer the traditional example of two wolves and a lamb voting on what to have for dinner.

The problem itself dates from time immemorial: the powerless have no way to compel the powerful. The only solution is to ensure that the powerful are on your side.

Comment author: Yvain 12 June 2009 05:20:09PM 4 points [-]

The problem is you've presented a fully general counterexample to all possible policies, including (apparently intentionally) a law against rape. Possibly also a fully general counterexample to ever being part of a group with other human beings (what if they're rapists?!?) I don't really see your point, other than trying to live up to your name.

Comment author: Annoyance 12 June 2009 04:56:31PM 1 point [-]

The other solution is government regulation.

There are more than two possible solutions to that problem, Yvain.

And "government regulation" isn't even a particularly good solution. What happens when the government is run by rapists?

Comment author: Yvain 12 June 2009 04:59:05PM 5 points [-]

What happens when the government is run by rapists?

Really, for maximal effect that comment should be followed by an "...OR DID I JUST BLOW YOUR MIND?!?!?!"

Comment author: RobinHanson 12 June 2009 03:42:23PM 4 points [-]

A common reason economists say noise regulation is not needed is that noise doesn't travel very far, most noise sources don't move, and so one can easily find the offending parties. So as long as it is clear who has the relevant property right (to make noise or to stop noise) they can plausibly make a deal to achieve the efficient outcome, at least if the problem is big enough.

Comment author: Yvain 12 June 2009 04:52:44PM *  10 points [-]

Silas has already come up with a good response, but...let's say this was implemented. And let's take the standard economic oversimplification of assuming mostly self-interested people.

And let's say I live in an apartment with six other people, one of whom is noisy. Five people are considerate and respectful of their neighbors, one is an inconsiderate asshole. I pay the asshole $100/month to do what everyone else does because they're a decent person. End result: being an inconsiderate asshole earns you $100/month. If you value fairness, this is already a bad outcome.

Now the other five people are upset, so they start making noise in the hope that I pay them $100. All this noise makes everyone unhappy, since everyone has at least some noise intolerance, and I don't have $500/month I can give away. I try to renegotiate the contract with the asshole, and he refuses. The other people can't back down, because they know this would ensure that they would never be respected as a bargaining partner again because even if I didn't pay them the money they would eventually stop making noise. The apartment becomes intolerably loud. This is an extremely bad outcome.

It becomes tempting to suggest that now everyone in the apartment make a deal, in which everyone who wants quiet pays a certain amount to everyone who wants to make noise, with the amount of money depending on how much each person believes in their individual preference. However, if you're quiet, there's a strong temptation to say you're actually loud in the hopes that other quiet people will buy you off. And if you're loud, there's a strong temptation to demand more money than your loudness is actually worth to you: that is, even if you don't really enjoy being loud, you should threaten to be really loud unless the quiet person agrees to pay you the absolute maximum amount ze can.

I read once about some people who tried paying kids for getting good grades (my memory is very hazy, I may be confusing some details of this study). They found that if they paid kids a small amount for good grades, their grades actually went down. When the kids weren't being paid, they were thinking in terms of "Do I have enough intrinsic motivation to want to do well?" and the answer was very often "yes". But when the kids were paid, they were thinking in terms of "is this amount I'm getting paid worth the effort of getting good grades", in which case the answer was very often "no". I think the same thing could happen here, leaving everyone worse off.

And finally, there's just plain ethical ramifications. Imagine an apartment with six people, some of whom are rapists. The rapists want to rape the non-rapists, and the non-rapists don't want to be raped. One solution would be that the non-rapists pay a certain amount of money to the rapists each month to incentivize them not to rape them. The other solution is government regulation. I think the government regulation solution comes a whole lot closer to our intuitive ethical conception of who has what obligations.

[another easy solution: simply have landowners or other nongovernment entities designate certain apartments or neighborhoods as "quiet zones" and others as "party zones". My old college did this with its dormitories, and it worked fine. Unfortunately, I have never seen this implemented in the real world with any sort of rigor.]

Typical Mind and Politics

46 Post author: Yvain 12 June 2009 12:28PM

Yesterday, in the The Terrible, Horrible, No Good Truth About Morality, Roko mentioned some good evidence that we develop an opinion first based on intuitions, and only later look for rational justifications. For example, people would claim incest was wrong because of worries like genetic defects or later harm, but continue to insist that incest was wrong even after all those worries had been taken away.

Roko's examples take advantage of universal human feelings like the incest taboo. But if people started out with opposite intuitions, then this same mechanism would produce opinions that people hold very strongly and are happy to support with as many reasons and facts as you please, but which are highly resistant to real debate or to contradicting evidence.

Sound familiar?

But to explain politics with this mechanism, we'd need an explanation for why people's intuitions differed to begin with. We've already discussed some such explanations - self-serving biases, influence from family and community, et cetera - but today I want to talk about another possibility.

A few weeks back, I was discussing harms with Bill Swift on Overcoming Bias. In particular, I was arguing that one situation in which there was an open-and-shut case for government restriction of private activity on private property was nuisance noise. I argued that if you were making noise on your property, and I could hear it on my property, that I was being harmed by your actions and that there was clearly just as much a case for government intervention here as if you were firing flaming arrows at me from your property. I fully expected Bill to agree that this was obviously true but to have some reason why he didn't think it applied to our particular disagreement.

Instead, to my absolute astonishment, Bill said that noise wasn't really a problem. He said he lived on a noisy property and had just stopped whining and gotten on with his life. I didn't really know how to react to this1, and ended up assuming either that he'd never lived in a really noisy place like I have, or that he was such a blighted ideologue that he was willing to completely contradict common sense in order to preserve his silly argument.

In other words, I was assuming the person I was debating was either astonishingly stupid or willfully evil. And when my thoughts tend in that direction, it usually means I'm missing something.

Luckily in this case I'd already written a long essay explaining my mistake in detail. In Generalizing From One Example,  I warned people against assuming everyone's mind is built the same way their own mind is. One particular example I gave was:

I can't deal with noise. If someone's being loud, I can't sleep, I can't study, I can't concentrate, I can't do anything except bang my head against the wall and hope they stop. I once had a noisy housemate. Whenever I asked her to keep it down, she told me I was being oversensitive and should just mellow out.


So it seems possible to me that I have an oversensitivity to noise and Bill has an undersensitivity to it. When someone around me is being noisy, my intuitions tell me this is extremely bad and needs to be stopped by any means necessary. And maybe Bill's intuitions tell him that this is a minor non-problem. I won't say that this is actually behind our disagreement on the issue - my guess is that Bill and I would disagree about government regulation of pollution from a factory as well - but I think it contributes and it makes our debate much less productive than it would have been otherwise.

Let me give an example of one place I think a mind difference *is* behind a political opinion. In Money, The Unit of Caring, Eliezer complained that people were too willing to donate time to charity, and too unwilling to donate money to charity. He gave the example of his own experience, where he felt terrible every time he gave away money, but didn't mind a time committment nearly as much. I fired back a response that this was completely foreign to me, because I am happy to give money to charity and often do it before I've even fully thought about what I'm doing, but will groan and make excuses whenever I'm asked to give away time. I also mentioned that this was a general tendency of mine: I have minimal aversion to monetary loss2, but wasting time makes me angry.

A few months ago, Barack Obama proposed a plan (which he later decided against) to make every high school and college student volunteer a certain amount of time to charity. Although I usually like Obama, I wrote an absolutely scathing essay about how unbearably bad a policy this was. It was a good essay, it convinced a number of people, and I still agree with most of the points in it. But...

...it was completely out of character for me. I'm the sort of person who heckles libertarians with "Stop whining and just pay your damn taxes!" Although I acknowledge that many government policies are inefficient, I tend to just note "Hmmm, that government policy is suboptimal, it would be an interesting mental puzzle to figure out how to fix it" rather than actually getting angry about it. This Obama proposal was kind of unique in the amount of antipathy it got from me.

So here's my theory. My brain is organized in such a way that I get minimal negative feelings at the idea of money being taken away from me. We can even localize this anatomically - studies show that the insula is the part responsible for sending a pain signal whenever the loss of money is considered. So let's say I have a less-than-normally-active insula in this case. And I get a stronger than normal pain signal from wasted time. This explains why I prefer to donate money than time to my favorite charity.

And it could also explain why I'm not a libertarian. One consequence of libertarianism is that you have every right to feel angry when you're taxed. But I don't feel angry, so the part of my brain that comes up with rational justifications for my feelings doesn't need to come up with a rational justification for why taxation is wrong. I do feel angry about being made to do extra work, so my brain adopted libertarian-type arguments in response to the community service proposal. I predict that if I lived in one of those feudal countries with a work levy rather than a tax, I'd be a libertarian, at least until the local knight heard my opinions and cut off my head.

And I don't mean to pick on libertarians. I know different people have completely different emotional responses to the idea of other people suffering. For example, I can't watch documentaries on (say) the awful lives on mine workers, because they make me too upset. Other people watch them, think they're great documentaries, and then spend the next hour talking about how upset it made them. And other people watch them and then ask what's for dinner. You think that affects people's opinions on socialism much?

Imagine a proposal to institute a tax that would raise money for some effort to help mine workers in some way. Upon hearing of it, different people would have an emotional burst of pain of a certain size at the thought of hearing of a tax, and an emotional burst of pain of a different size at the thought of considering the mine workers. Neither of these bursts of pain would be proportional to the actual size of the problem as measured in some sort of ideal utilon currency (note especially scope insensitivity). But the brain very often makes decisions by comparing those two bursts of pain (see How We Decide or just the insula article above) and then comes up with reasons for the decision. So all the important issues like economic freedom and labor policy and maximizing utility and suchwhat get subordinated to whether you're secreting more neurotransmitters in response to money loss or images of sad coal miners.

If this theory were true, we would expect to find neurological differences in people of different political opinions. Ta da! A long list of neurological findings that differ in liberals and conservatives. Linking the startle reflex and the disgust reaction to the policies favored by these groups is left as a (very easy) exercise for the reader3.

This may require some moderation of our political opinions on issues where we think we're far from the neurological norm. For example, I am no longer so confident that noise is such a big problem for everyone that we would all be better off if there were strict regulations on it. But I hope Bill will consider that some people may be so sensitive to noise that not everyone can just shrug it off, and so there may be a case for at least some regulation of it. Likewise, even though I don't mind taxes too much, if my goal is a society where most people are happy I need to consider that a higher tax rate will decrease other people's happiness much more quickly than it decreases mine.

Other than that, it's just a general message of pessimism. If people's political opinions come partly from unchangeable anatomy, it makes the program of overcoming bias in politics a lot harder, and the possibility of coming up with arguments good enough to change someone else's opinion even more remote.

Footnotes

1) I am suitably ashamed of my appeal to pathos; my only defense is that it is entirely true, that I have only just finished moving, and that this post is hopefully a more appropriate response.

2) Actually, it's more complicated than this, because I agonize over spending money when shopping. I seem to use different thought processes for normal budgeting, and I expect there are many processes going on more complex than just high versus low aversion to money loss.

3) Possibly too easy. It's easy to go from that data to an explanation of why conservatives worry more about terrorism, but then why don't they also worry more about global warming?

A Parable On Obsolete Ideologies

113 Post author: Yvain 13 May 2009 10:51PM

Followup to:  Yudkowsky and Frank on Religious Experience, Yudkowksy and Frank On Religious Experience Pt 2
With sincere apologies to: Mike Godwin

You are General Eisenhower. It is 1945. The Allies have just triumphantly liberated Berlin. As the remaining leaders of the old regime are being tried and executed, it begins to become apparent just how vile and despicable the Third Reich truly was.

In the midst of the chaos, a group of German leaders come to you with a proposal. Nazism, they admit, was completely wrong. Its racist ideology was false and its consequences were horrific. However, in the bleak poverty of post-war Germany, people need to keep united somehow. They need something to believe in. And a whole generation of them have been raised on Nazi ideology and symbolism. Why not take advantage of the national unity Nazism provides while discarding all the racist baggage? "Make it so," you say.

The swastikas hanging from every boulevard stay up, but now they represent "traditional values" and even "peace". Big pictures of Hitler still hang in every government office, not because Hitler was right about racial purity, but because he represents the desire for spiritual purity inside all of us, and the desire to create a better society by any means necessary. It's still acceptable to shout "KILL ALL THE JEWS AND GYPSIES AND HOMOSEXUALS!" in public places, but only because everyone realizes that Hitler meant "Jews" as a metaphor for "greed", "gypsies" as a metaphor for "superstition", and "homosexuals" as a metaphor for "lust", and so what he really meant is that you need to kill the greed, lust, and superstition in your own heart. Good Nazis love real, physical Jews! Some Jews even choose to join the Party, inspired by their principled stand against spiritual evil.

The Hitler Youth remains, but it's become more or less a German version of the Boy Scouts. The Party infrastructure remains, but only as a group of spiritual advisors helping people fight the untermenschen in their own soul. They suggest that, during times of trouble, people look to Mein Kampf for inspiration. If they open to a sentence like "The Aryan race shall conquer all in its path", then they can interpret "the Aryan race" to mean "righteous people", and the sentence is really just saying that good people can do anything if they set their minds to it. Isn't that lovely?

Soon, "Nazi" comes to just be a synonym for "good person". If anyone's not a member of the Nazi Party, everyone immediately becomes suspicious. Why is she against exterminating greed, lust, and superstition from her soul? Does she really not believe good people can do anything if they set their minds to it? Why does he oppose caring for your aging parents? We definitely can't trust him with high political office.

It is four years later. Soon, the occupation will end, and Germany will become an independent country once again. The Soviets have already taken East Germany and turned it Communist. As the de facto ruler of West Germany, its fate is in your hands. You ask your two most trusted subordinates for advice.

First, Colonel F gives his suggestion. It is vital that you order the preservation of the Nazi ideology so that Germany remains strong. After all, the Germans will need to stay united as a people in order to survive the inevitable struggle with the Soviets. If Nazism collapsed, then people would lose everything that connects them together, and become dispirited. The beautiful poetry of Mein Kampf speaks to something deep in the soul of every German, and if the Allies try to eradicate that just because they disagree with one outdated interpretation of the text, they will have removed meaning from the lives of millions of people all in the name of some sort of misguided desire to take everything absolutely literally all the time.

Your other trusted subordinate, Colonel Y, disagrees. He thinks that Mein Kampf may have some rousing passages, but that there's no special reason it has a unique ability to impart meaning to people other than that everyone believes it does. Not only that, but the actual contents of Mein Kampf are repulsive. Sure, if you make an extraordinary effort to gloss over or reinterpret the repulsive passages, you can do it, but this is more trouble than it is worth and might very well leave some lingering mental poison behind. Germany should completely lose all the baggage of Nazism and replace it with a completely democratic society that has no causal linkage whatsoever to its bloody past.

Colonel F objects. He hopes you don't just immediately side with Colonel Y just because the question includes the word "Nazi". Condemning Nazism is an obvious applause light, but a political decision of this magnitude requires a more carefully thought-out decision. After all, Nazism has been purged of its most objectionable elements, and the Germans really do seem to like it and draw a richer life from it. Colonel Y needs to have a better reason his personal distaste for an ideology because of past history in order to take it away from them.

Colonel Y thinks for a moment, then begins speaking. You have noticed, he says, that the new German society also has a lot of normal, "full-strength" Nazis around. The "reformed" Nazis occasionally denounce these people, and accuse them of misinterpreting Hitler's words, but they don't seem nearly as offended by the "full-strength" Nazis as they are by the idea of people who reject Nazism completely.

Might the existence of "reformed" Nazis, he asks, enable "full-strength" Nazis to become more powerful and influential? He thinks it might. It becomes impossible to condemn "full-strength" Nazis for worshipping a horrible figure like Hitler, or adoring a horrible book like Mein Kampf, when they're doing the same thing themselves. At worst, they can just say the others are misinterpreting it a little. And it will be very difficult to make this argument, because all evidence suggests that in fact it's the "full-strength" Nazis who are following Hitler's original intent and the true meaning of Mein Kampf, and the "reformed" Nazis who have reinterpreted it for political reasons. Assuming the idea of not being a Nazi at all remains socially beyond the pale, intellectually honest people will feel a strong pull towards "full-strength" Nazism.

Even if the "reformed" Nazis accept all moderate liberal practices considered reasonable today, he says, their ideology might still cause trouble later. Today, in 1945, mixed race marriage is still considered taboo by most liberal societies, including the United States. The re-interpreters of Mein Kampf have decided that, although "kill all the Jews" is clearly metaphorical, "never mix races" is meant literally. If other nations began legalizing mixed race marriage in the years to come, Party members will preach to the faithful that it is an abomination, and can even point to the verse in Mein Kampf that said so. It's utterly plausible that a "reformed" Nazi Germany may go on forbidding mixed race marriage much longer than surrounding countries. Even if Party leaders eventually bow to pressure and change their interpretation, the Party will always exist as a force opposing racial equality and social justice until the last possible moment.

And, he theorizes, there could be even deeper subconscious influences. He explains that people often process ideas and morals in ways that are only tangentially linked to specific facts and decisions. Instead, we tend to conflate things into huge, fuzzy concepts and assign "good" and "bad" tags to them. Saying "Jews are bad, but this doesn't apply to actual specific Jews" is the sort of thing the brain isn't very good at. At best, it will end out with the sort of forced politeness a person who's trying very hard not be racist shows around black people. As soon as we assign a good feeling to the broad idea of "Nazism", that reflects at least a little on everything Nazism stands for, everything Nazism ever has stood for, and every person who identifies as a Nazi.

He has read other essays that discuss the ability of connotations to warp thinking. Imagine you're taught things like "untermenschen like Jews and Gypsies are people too, and should be treated equally." The content of this opinion is perfectly fine. Unfortunately, it creates a category called "untermenschen" with a bad connotation and sticks Jews and Gypsies into it. Once you have accepted that Jews and Gypsies comprise a different category, even if that category is "people who are exactly like the rest of us except for being in this category here", three-quarters of the damage is already done. Here the Colonel sighs, and reminds you of the discrimination faced by wiggins in the modern military.

And (he adds) won't someone please think of the children? They're not very good at metaphor, they trust almost anything they hear, and they form a scaffolding of belief that later life can only edit, not demolish and rebuild. If someone was scared of ghosts as a child, they may not believe in ghosts now, but they're going to have some visceral reaction to them. Imagine telling a child "We should kill everyone in the lesser races" five times a day, on the assumption that once they're a teenager they'll understand what a "figurative" means and it'll all be okay.

He closes by telling you that he's not at all convinced that whatever metaphors the Nazis reinterpret Mein Kampf to mean aren't going to be damaging in themselves. After all, these metaphors will have been invented by Nazis, who are not exactly known for choosing the best moral lessons. What if "kill all lesser races" gets reinterpreted to "have no tolerance for anything that is less than perfect"? This sounds sort of like a good moral lesson, until people start preaching that it means we should lock up gay people, because homosexuality is an "imperfection". That, he says, is the sort of thing that happens when you get your morality from cliched maxims taken by drawing vapid conclusions from despicably evil works of literature.

So, the Colonel concludes, if you really want the German people to be peaceful and moral, you really have no choice but to nip this growing "reformed Nazi" movement in the bud. Colonel F has made some good points about respecting the Germans' culture, but doing so would make it difficult to eradicate their existing racist ideas, bias their younger generation towards habits of thought that encourage future racism, create a strong regressive tendency in their society, and yoke them to poorly fashioned moral arguments.

And, he finishes, he doesn't really think Nazism is that necessary for Germany to survive. Even in some crazy alternate universe where the Allies had immediately cracked down on Nazism as soon as they captured Berlin, yea, even in the absurd case where Germany immediately switched to a completely democratic society that condemned everything remotely associated with Nazism as evil and even banned swastikas and pictures of Hitler from even being displayed - even in that universe, Germans would keep a strong cultural identity and find new symbols of their patriotism.

Ridiculous, Colonel F objects! In such a universe, the Germans would be left adrift without the anchor of tradition, and immediately be taken over by the Soviets.

Colonel Y just smiles enigmatically. You are reminded of the time he first appeared at your command tent, during the middle of an unnatural thunderstorm, with a copy of Hugh Everett's The Theory of the Universal Wave Function tucked under one arm. You shudder, shake your head, and drag yourself back to the present.

So, General, what is your decision?

Comment author: rhollerith 30 May 2009 11:45:09AM *  1 point [-]

Status seekers probably greatly outnumber true altruists.

But you should tend to keep the status seekers out of positions of great responsibility IMHO even if doing so greatly reduces the total number of volunteers working on existential risks.

My tentative belief that status seekers will not do as good a job BTW stems from (1) first-hand observation and second-hand observation of long-term personal performance as a function of personal motivation in domains such as science-learning, programming, management and politics and (2) a result from social psychology that intrinsic reinforcers provide more reliable motivation than extrinsic reinforcers (for more about which, google "Punished by Rewards").

The last thing the future light cone needs is for existential-risk activism to become the next big thing in how to show prospective friends and prospective lovers how cool you are.

Comment author: Yvain 30 May 2009 01:13:52PM 4 points [-]

A lot more people talk about existential risks, often in a very animated way, than do anything about them.

I think probably the vast majority of people interested in existential risk want to signify both that they are good caring people, and that they are hard-headed intelligent rationalists and not the sort of muddled peace-and-love types who would go around waving "FUR IS MURDER" signs.

Probably doesn't actually work as far as getting friends and lovers is concerned, but it's a good self-signal.

Comment author: Yvain 15 May 2009 11:06:13AM *  151 points [-]

This is a beautiful example of politics at work.

Cheerios claims on its box that it can "lower your cholesterol four percent in six weeks". This is false. It is based on a "study" sponsored by General Mills where subjects took more than half their daily calories from Cheerios (apparently they ate nothing but Cheerios for two of their three daily meals). No one eating Cheerios in anything resembling a normal way would get close to this effect; therefore, it is false and misleading advertising. If General Mills wants to market Cheerios as a drug, it needs to meet the normal standards for drug evidence, and it doesn't.

So far, so good. Either you approve of the FDA's decision and think it's important to hold cereal companies to a high level of accuracy, or you think they should relax their standards and allow more leeway to food advertisers. Either one would be a legitimate response. But look at what happens:

A few sources correctly title the story, eg "Cheerios Aren't A Drug, FDA Says". The majority choose to go the other way and title it something more inflammatory like "Popular Cereal Is A Drug, US Food Watchdog Says", which is of course the opposite of what it said. It's the inflammatory outrageous headlines that get put on blogs and Reddit (now reworded further to "WTF? FDA says Cheerios are a drug")

Then all the usual suspects take the mistitled blown-out-of-proportion story and look to see whether or not this supports their preferred political narrative. For example:

Independent Institute (a libertarian think tank):

"The Obama administration blunders onward in its “progressive” (i.e., authoritarian) absurdities."

Freedom News Blitz ("The News Freedom Lovers Devour"):

"Do you get the feeling that since we are experiencing a severe recession, the Food and Drug Administration is running out of honest businesses to harass and persecute?...Certainly, it appears the old virtues of free enterprise, hard work, self-discipline, saving money, weighing the short-term consequences against the long-term consequences, etc. are unpalatable to the vast majority of people. So they allow the false promises of politicians, civic leaders and pseudo-intellectuals to mislead them.

So yes, we went from "please don't use poorly designed made-up studies to make spurious medical claims" to "hard work is unpalatable to the vast majority of people" in two steps. Remarkable, ne?

And then the media goes into one of its periodic vicious feedback loops and started reporting on the reporting, adding a little more liberal bent at each pass. From Reuters, here's Cheerios, Cereal of Liberty. The article starts with "Disputes over food-label claims are always political" and quickly moves onto "But the current, insane iteration of the American right has walked several steps past the crazy line...For them, wholesome, "American" foods are a-OK. Eurocommie foods are right out."

Three steps and now we're at "Eurocommie."

In the last phase of the decline from "reasonable question about cholesterol lowering properties of cereal" to "complete proof humankind as a species is doomed", someone opens the floodgates to Mordor and a horde of semi-human blog commenters swarm out, ready to add their "opinion" to the "discussion". From here and here:

"Ha - I guess the makers of the liver corroding, memory erasing cholesterol drugs, must have whined about this to their best friends, (the guys they pay off to legalize these poisons) the FDA."

"Sorry mister.g but you folks are delusional in the USA. You buy into everything and I mean everything. That is why you are the sickest, poorest industrialized economy going and you can inject GOLD into dog poop but it will still be dog poop. If you think it's worth eating to get to that golden nugget in the center you go right ahead. Unbelievable."

"Unapproved though (non-Obama worship) is not approved by your government overlords. Americas are too stupid to understand what is a cereal and what is a drug. You will be eliminated."

Four steps from "Don't lie about cholesterol on your cereal box, please," to "Obama will kill everyone who disagrees with him."

In response to comment by taw on Survey Results
Comment author: mattnewport 14 May 2009 06:14:39PM 6 points [-]

I don't know where you've been finding this argument but it's hardly representative of a good argument for libertarianism. I grew up in Europe (well, the UK, which is kind of Europe) with Labour voting parents and grandparents with fairly socialist views and considered myself a socialist into my early 20s. Weak arguments like these wouldn't have been enough to convert me to a generally libertarian worldview.

I had a similar caricature of the views of supporters of the free market (back when I didn't even know the term libertarian) but learning more about economics and being confronted with evidence of better outcomes in freer economies, together with learning that few serious economists (or libertarians) believe in perfectly efficient markets and learning about Public Choice Theory were key in changing my political views.

Key to the economic arguments for libertarianism is the idea that incentives matter and that the incentives facing actors in a free market tend to be far less perverse than those facing politicians or employees of state run monopolies.

The moral arguments stem largely from a view that personal freedom is a high moral value and that the evidentiary bar should be set very high for any demonstration of harm to justify restriction of individual freedoms. That tendency seems to be correlated with certain personality types according to some research and the crossover between libertarians and progressives/liberals on social issues seems to be as much a factor of personal values as of consequentialist reasoning.

And being fairly familiar with UK politics (less so with European politics in other countries) the idea that European politics pick policies based on 'what is estimated to work best' strikes me as pretty laughable.

In response to comment by mattnewport on Survey Results
Comment author: Yvain 14 May 2009 06:49:31PM 3 points [-]

Thanks, Matt. You're providing some interesting points in a direction I hadn't heard much about before.

Do you think most libertarians believe that regulation by a responsible, intelligent, benevolent government would improve society, but that we simply don't have a government we can trust that much? Or do you think they believe that any government intervention is likely to have adverse effects no matter how well-planned it is?

Comment author: newerspeak 14 May 2009 11:15:35AM *  2 points [-]

Do you go looking for merit-worthy religious apologetics?

Yes. Diagnosing the faults in Alvin Plantinga's reasoning is important. Am I to understand you'd prefer a frank exchange of views with Jerry Falwell?

That said, do you know of any libertarian arguments that do not assume either 1) economic freedom as the primary terminal value or 2) assume the efficiency of real-world markets? Both are unwarranted assumptions that seem to underlie many libertarian arguments I've seen.

Yes. I included one such argument in the post you just replied to. I quote myself:

One interestding claim [of policy libertarianism]: State actors are (made up of) people who are subject to the same irrational biases and collective stupidity as market actors, and often have perverse incentive structures as well."

In other words, government decision-makers (i.e. bureaucrats) have just as much trouble integrating new information, violating social norms, and admitting error as consumers or decision-makers for firms, but bureaucrats are also subject to perverse incentives, regulatory capture, etc.

The implied primary terminal value here is welfare-maximization, according to some material standard that I'm assuming we could agree on, given that we're both here. No specific claim about the efficiency of markets is made. A fortiori, the argument derives some of its strength from the acknowledgment of certain deviations from rational behavior that (once again) we both presumably know about, because we're both here.

In response to comment by newerspeak on Survey Results
Comment author: Yvain 14 May 2009 12:28:24PM *  6 points [-]

One interesting claim [of policy libertarianism]: State actors are (made up of) people who are subject to the same irrational biases and collective stupidity as market actors, and often have perverse incentive structures as well."

My main complaint with this argument is that it should be empirically testable. You can implement regulatory scheme X in Area A, and no regulatory scheme in Area B, and see which produces better results. For example, ban all cancer treatments that top doctors agree are useless and dangerous in Area A, keep all treatments legal in Area B, and see which area has higher mortality among cancer patients.

Many libertarians I know have absolutely no interest in doing this, and don't even like talking about the term "regulatory scheme X" because they prefer to lump all possible regulatory schemes together and judge them on the merit of the first one that comes to mind (this is also a problem with many socialists, for the opposite reason).

I don't know much about economics, but I do know a bit about public health policy, and the people in charge of that are sometimes very good about using studies to determine whether their government interventions are an overall improvement over the no-intervention case (obvious exception: the FDA, which is very good at running studies, but very bad at running the right studies and doing sane cost-benefit analysis). When these studies show positive results at relatively low cost, a truly consequentialist libertarian ought to admit government regulation has been effective in that case. Instead, they tend to dismiss it as a fluke or start talking about some case where government regulation isn't effective.

I think the great error in this whole debate is framing it as a conflict between socialists (who supposedly ought to think all government interventions are great) and libertarians (who supposedly ought to think all government interventions are terrible). In reality, some of these will work and some of these won't. I'd rather people started paying more attention to which were which than become crusaders for bigger or smaller government. I think "Government regulation is bad" (or "is good") is approximately the same kind of sentence as "Islam is a religion of peace".

In response to comment by taw on Survey Results
Comment author: Douglas_Knight 13 May 2009 05:34:49PM -1 points [-]

Certainly most libertarians care about processes, or at most about results very similar to the processes, but this is a biased sample.

Most ideologies are about process and uninterested in evidence about consequences, but that doesn't mean that people who the term "libertarian" are ideologues. One cost of using the term is appearing to be an ideologue. For this reason, I refuse to reliquish the term "liberal" to the modern liberals. But I think that taw is poisoning the discourse, making it worse than it already is. It's a pretty common tactic to paint anyone outside the mainstream as an ideologue.

Comment author: Yvain 13 May 2009 11:00:58PM *  6 points [-]

Oh, hey, we have data!

According to crosstabs, of our fifteen deontologists, four were libertarian, four were liberal, four were socialist, two were conservative, and one didn't list political views. That means deontologists were slightly less likely to be libertarian than the average person.

(deontologists were much more likely to be conservative than the average person, but I can't draw too many conclusions from that because there was such a small sample size of deontologists and conservatives.)

I admit I didn't expect that result. I think it's because the really, really loud obnoxious libertarians like Objectivists are all deontologists. But I don't think this site has a lot of those. I would be curious what would happen if we polled the reader based of lewrockwell.com

[EDIT: Also, an overwhelming majority of those who said they didn't believe in morality were libertarians. Wonder what that means.].

A Parable On Obsolete Ideologies

113 Post author: Yvain 13 May 2009 10:51PM

Followup to:  Yudkowsky and Frank on Religious Experience, Yudkowksy and Frank On Religious Experience Pt 2
With sincere apologies to: Mike Godwin

You are General Eisenhower. It is 1945. The Allies have just triumphantly liberated Berlin. As the remaining leaders of the old regime are being tried and executed, it begins to become apparent just how vile and despicable the Third Reich truly was.

In the midst of the chaos, a group of German leaders come to you with a proposal. Nazism, they admit, was completely wrong. Its racist ideology was false and its consequences were horrific. However, in the bleak poverty of post-war Germany, people need to keep united somehow. They need something to believe in. And a whole generation of them have been raised on Nazi ideology and symbolism. Why not take advantage of the national unity Nazism provides while discarding all the racist baggage? "Make it so," you say.

The swastikas hanging from every boulevard stay up, but now they represent "traditional values" and even "peace". Big pictures of Hitler still hang in every government office, not because Hitler was right about racial purity, but because he represents the desire for spiritual purity inside all of us, and the desire to create a better society by any means necessary. It's still acceptable to shout "KILL ALL THE JEWS AND GYPSIES AND HOMOSEXUALS!" in public places, but only because everyone realizes that Hitler meant "Jews" as a metaphor for "greed", "gypsies" as a metaphor for "superstition", and "homosexuals" as a metaphor for "lust", and so what he really meant is that you need to kill the greed, lust, and superstition in your own heart. Good Nazis love real, physical Jews! Some Jews even choose to join the Party, inspired by their principled stand against spiritual evil.

The Hitler Youth remains, but it's become more or less a German version of the Boy Scouts. The Party infrastructure remains, but only as a group of spiritual advisors helping people fight the untermenschen in their own soul. They suggest that, during times of trouble, people look to Mein Kampf for inspiration. If they open to a sentence like "The Aryan race shall conquer all in its path", then they can interpret "the Aryan race" to mean "righteous people", and the sentence is really just saying that good people can do anything if they set their minds to it. Isn't that lovely?

Soon, "Nazi" comes to just be a synonym for "good person". If anyone's not a member of the Nazi Party, everyone immediately becomes suspicious. Why is she against exterminating greed, lust, and superstition from her soul? Does she really not believe good people can do anything if they set their minds to it? Why does he oppose caring for your aging parents? We definitely can't trust him with high political office.

It is four years later. Soon, the occupation will end, and Germany will become an independent country once again. The Soviets have already taken East Germany and turned it Communist. As the de facto ruler of West Germany, its fate is in your hands. You ask your two most trusted subordinates for advice.

First, Colonel F gives his suggestion. It is vital that you order the preservation of the Nazi ideology so that Germany remains strong. After all, the Germans will need to stay united as a people in order to survive the inevitable struggle with the Soviets. If Nazism collapsed, then people would lose everything that connects them together, and become dispirited. The beautiful poetry of Mein Kampf speaks to something deep in the soul of every German, and if the Allies try to eradicate that just because they disagree with one outdated interpretation of the text, they will have removed meaning from the lives of millions of people all in the name of some sort of misguided desire to take everything absolutely literally all the time.

Your other trusted subordinate, Colonel Y, disagrees. He thinks that Mein Kampf may have some rousing passages, but that there's no special reason it has a unique ability to impart meaning to people other than that everyone believes it does. Not only that, but the actual contents of Mein Kampf are repulsive. Sure, if you make an extraordinary effort to gloss over or reinterpret the repulsive passages, you can do it, but this is more trouble than it is worth and might very well leave some lingering mental poison behind. Germany should completely lose all the baggage of Nazism and replace it with a completely democratic society that has no causal linkage whatsoever to its bloody past.

Colonel F objects. He hopes you don't just immediately side with Colonel Y just because the question includes the word "Nazi". Condemning Nazism is an obvious applause light, but a political decision of this magnitude requires a more carefully thought-out decision. After all, Nazism has been purged of its most objectionable elements, and the Germans really do seem to like it and draw a richer life from it. Colonel Y needs to have a better reason his personal distaste for an ideology because of past history in order to take it away from them.

Colonel Y thinks for a moment, then begins speaking. You have noticed, he says, that the new German society also has a lot of normal, "full-strength" Nazis around. The "reformed" Nazis occasionally denounce these people, and accuse them of misinterpreting Hitler's words, but they don't seem nearly as offended by the "full-strength" Nazis as they are by the idea of people who reject Nazism completely.

Might the existence of "reformed" Nazis, he asks, enable "full-strength" Nazis to become more powerful and influential? He thinks it might. It becomes impossible to condemn "full-strength" Nazis for worshipping a horrible figure like Hitler, or adoring a horrible book like Mein Kampf, when they're doing the same thing themselves. At worst, they can just say the others are misinterpreting it a little. And it will be very difficult to make this argument, because all evidence suggests that in fact it's the "full-strength" Nazis who are following Hitler's original intent and the true meaning of Mein Kampf, and the "reformed" Nazis who have reinterpreted it for political reasons. Assuming the idea of not being a Nazi at all remains socially beyond the pale, intellectually honest people will feel a strong pull towards "full-strength" Nazism.

Even if the "reformed" Nazis accept all moderate liberal practices considered reasonable today, he says, their ideology might still cause trouble later. Today, in 1945, mixed race marriage is still considered taboo by most liberal societies, including the United States. The re-interpreters of Mein Kampf have decided that, although "kill all the Jews" is clearly metaphorical, "never mix races" is meant literally. If other nations began legalizing mixed race marriage in the years to come, Party members will preach to the faithful that it is an abomination, and can even point to the verse in Mein Kampf that said so. It's utterly plausible that a "reformed" Nazi Germany may go on forbidding mixed race marriage much longer than surrounding countries. Even if Party leaders eventually bow to pressure and change their interpretation, the Party will always exist as a force opposing racial equality and social justice until the last possible moment.

And, he theorizes, there could be even deeper subconscious influences. He explains that people often process ideas and morals in ways that are only tangentially linked to specific facts and decisions. Instead, we tend to conflate things into huge, fuzzy concepts and assign "good" and "bad" tags to them. Saying "Jews are bad, but this doesn't apply to actual specific Jews" is the sort of thing the brain isn't very good at. At best, it will end out with the sort of forced politeness a person who's trying very hard not be racist shows around black people. As soon as we assign a good feeling to the broad idea of "Nazism", that reflects at least a little on everything Nazism stands for, everything Nazism ever has stood for, and every person who identifies as a Nazi.

He has read other essays that discuss the ability of connotations to warp thinking. Imagine you're taught things like "untermenschen like Jews and Gypsies are people too, and should be treated equally." The content of this opinion is perfectly fine. Unfortunately, it creates a category called "untermenschen" with a bad connotation and sticks Jews and Gypsies into it. Once you have accepted that Jews and Gypsies comprise a different category, even if that category is "people who are exactly like the rest of us except for being in this category here", three-quarters of the damage is already done. Here the Colonel sighs, and reminds you of the discrimination faced by wiggins in the modern military.

And (he adds) won't someone please think of the children? They're not very good at metaphor, they trust almost anything they hear, and they form a scaffolding of belief that later life can only edit, not demolish and rebuild. If someone was scared of ghosts as a child, they may not believe in ghosts now, but they're going to have some visceral reaction to them. Imagine telling a child "We should kill everyone in the lesser races" five times a day, on the assumption that once they're a teenager they'll understand what a "figurative" means and it'll all be okay.

He closes by telling you that he's not at all convinced that whatever metaphors the Nazis reinterpret Mein Kampf to mean aren't going to be damaging in themselves. After all, these metaphors will have been invented by Nazis, who are not exactly known for choosing the best moral lessons. What if "kill all lesser races" gets reinterpreted to "have no tolerance for anything that is less than perfect"? This sounds sort of like a good moral lesson, until people start preaching that it means we should lock up gay people, because homosexuality is an "imperfection". That, he says, is the sort of thing that happens when you get your morality from cliched maxims taken by drawing vapid conclusions from despicably evil works of literature.

So, the Colonel concludes, if you really want the German people to be peaceful and moral, you really have no choice but to nip this growing "reformed Nazi" movement in the bud. Colonel F has made some good points about respecting the Germans' culture, but doing so would make it difficult to eradicate their existing racist ideas, bias their younger generation towards habits of thought that encourage future racism, create a strong regressive tendency in their society, and yoke them to poorly fashioned moral arguments.

And, he finishes, he doesn't really think Nazism is that necessary for Germany to survive. Even in some crazy alternate universe where the Allies had immediately cracked down on Nazism as soon as they captured Berlin, yea, even in the absurd case where Germany immediately switched to a completely democratic society that condemned everything remotely associated with Nazism as evil and even banned swastikas and pictures of Hitler from even being displayed - even in that universe, Germans would keep a strong cultural identity and find new symbols of their patriotism.

Ridiculous, Colonel F objects! In such a universe, the Germans would be left adrift without the anchor of tradition, and immediately be taken over by the Soviets.

Colonel Y just smiles enigmatically. You are reminded of the time he first appeared at your command tent, during the middle of an unnatural thunderstorm, with a copy of Hugh Everett's The Theory of the Universal Wave Function tucked under one arm. You shudder, shake your head, and drag yourself back to the present.

So, General, what is your decision?

In response to Survey Results
Comment author: Unnamed 13 May 2009 03:29:16AM *  4 points [-]

There was a .453 correlation between this number and actual IQ; that is, 45% of the variance in how likely you thought you were to have a higher-than-average IQ could be explained by your actual IQ.

Correlation is r and percent of variance explained is r^2, so I think that should be 21% rather than 45%. There's also a typo where you say ".5 level" and presumably mean .05.

In response to comment by Unnamed on Survey Results
Comment author: Yvain 13 May 2009 06:40:15AM 1 point [-]

Thanks. Edited out.

In response to Survey Results
Comment author: John_Maxwell_IV 13 May 2009 04:57:13AM *  5 points [-]

Why the heck is the average stated probability of a creator god greater than the average stated probability of something supernatural existing? Or did a team of scientists in a parallel universe count as a creator god?

Comment author: Yvain 13 May 2009 06:39:46AM 6 points [-]

I think the supernatural question was phrased as something supernatural happening within the universe. So in a deist perspective, if God created the universe and then went away, that would qualify for creator god but not supernatural.

In response to Survey Results
Comment author: dfranke 12 May 2009 11:50:31PM 3 points [-]

If you intend to hide something by shuffling the results, it's probably also a good idea to remove the "timestamp" column :-)

In response to comment by dfranke on Survey Results
Comment author: Yvain 13 May 2009 06:38:41AM 0 points [-]

...right. Done.

Survey Results

48 Post author: Yvain 12 May 2009 10:09PM

Followup to: Excuse Me, Would You Like to Take a Survey?, Return of the Survey

Thank you to everyone who took the Less Wrong survey. I've calculated some results out on SPSS, and I've uploaded the data for anyone who wants it. I removed twelve people who wanted to remain private, removed a few people's karma upon request, and re-sorted the results so you can't figure out that the first person on the spreadsheet was the first person to post "I took it" on the comments thread and so on. Warning: you will probably not get exactly the same results as me, because a lot of people gave poor, barely comprehensible write in answers, which I tried to round off to the nearest bin.

Download the spreadsheet (right now it's in .xls format)

I am not a statistician, although I occasionally have to use statistics for various things, and I will gladly accept corrections for anything I've done wrong. Any Bayesian purists may wish to avert their eyes, as the whole analysis is frequentist. What can I say? I get SPSS software and training free and I don't like rejecting free stuff. The write-up below is missing answers to a few questions that I couldn't figure out how to analyze properly; anyone who cares about them enough can look at the raw data and try it themselves. Results under the cut.

Out of 166 respondees:

160 (96.4%) were male, 5 (3%) were female, and one chose not to reveal their gender.

The mean age was 27.16, the median was 25, and the SD was 7.68. The youngest person was 16, and the oldest was 60. Quartiles were <22, 22-25, 25-30, and >30.

Of the 158 of us who disclosed our race, 148 were white (93.6%), 6 were Asian, 1 was Black, 2 were Hispanic, and one cast a write-in vote for Middle Eastern. Judging by the number who put "Hinduism" as their family religion, most of those Asians seem to be Indians.

Of the 165 of us who gave readable relationship information, 55 (33.3%) are single and looking, 40 (24.2%) are single but not looking, 40 (24.2%) are in a relationship, 29 (17.6%) are married, and 1 is divorced.

Only 138 gave readable political information (those of you who refused to identify with any party and instead sent me manifestos, thank you for enlightening me, but I was unfortunately unable to do statistics on them). We have 62 (45%) libertarians, 53 (38.4%) liberals, 17 (12.3%) socialists, 6 (4.3%) conservatives, and not one person willing to own up to being a commie.

Of the 164 people who gave readable religious information, 134 (81.7%) were atheists and not spiritual; 5 other atheists described themselves as "spiritual". Counting deists and pantheists, we had 11 believers in a supreme being (6.7%), of whom 2 were deist/pantheist, 2 were lukewarm theists, and 6 were committed theists. 14 of us (8.5%) were agnostic.

53 of us were raised in families of "about average religiousity" (31.9%). 24 (14.5%) were from extremely religious families, 45 (27.1%) from nonreligious families, and 9 (5.4%) from explicitly atheist families. 30 (18.1%) were from families less religious than average. The remainder wrote in some hard to categorize responses, like an atheist father and religious mother, or vice versa.

Of the 106 of us who listed our family's religious background, 92 (87%) were Christian. Of the Christians, 29 (31.5% of Christians) described their backgrounds as Catholic, 30 (32.6% of Christians) described it as Protestant, and the rest gave various hard-to-classify denominations or simply described themselves as "Christian". There were also 9 Jews, 3 Hindus, 1 Muslim, and one New Ager.

I didn't run the "how much of Overcoming Bias have you read" question so well, and people ended up responding things like "Oh, most of it", which are again hard to average. After interpreting things extremely liberally and unscientifically ("most" was estimated as 75%, "a bit" was estimated at 25%, et cetera) I got that the average LWer has read about half of OB, with a slight tendency to read more of Eliezer's posts than Robin's.

Average time in the OB/LW community was 13.6 ± 9.2 months. Average time spent on the site per day was 30.7 ± 30.4 minutes.

IQs (warning: self-reported numbers for notoriously hard-to-measure statistic) ranged from 120 to 180. The mean was 145.88, median was 141.50, and SD was 14.02. Quartiles were <133, 133-141.5, 141.5-155, and >155.

77 people were willing to go out on a limb and guess whether their IQ would be above the median or not. The mean confidence level was 54.4, and the median confidence level was 55 - which shows a remarkable lack of self-promoting bias. The quartiles were <40, 40-55, 55-70, >70. There was a .453 correlation between this number and actual IQ. This number was significant at the <.001 level.

Probability of Many Worlds being more or less correct (given as mean, median, SD; all probabilities in percentage format): 55.65, 65, 32.9.

Probability of aliens in the observable Universe: 70.3, 90, 35.7.

Probability of aliens in our galaxy: 40.9, 35, 38.5. Notice the huge standard deviations here; the alien questions were remarkable both for the high number of people who put answers above 99.9, and the high number of people who put answers below 0.1. My guess: people who read about The Great Filter versus those who didn't.

Probability of some revealed religion being true: 3.8, 0, 12.6.

Probability of some Creator God: 4.2, 0, 14.6.

Probability of something supernatural existing: 4.1, 0, 12.8.

Probability of an average person cryonically frozen today being successfully revived: 22.3, 10, 26.2.

Probability of anti-agathic drugs allowing the current generation to live beyond 1000: 29.2, 20, 30.8.

Probability that we live in a simulation: 16.9, 5, 23.7.

Probability of anthropic global warming: 69.4, 80, 27.8.

Probability that we make it to 2100 without a catastrophe killing >90% of us: 73.1, 80, 24.6.

When asked to determine a year in which the Singularity might take place, the mean guess was 9,899 AD, but this is only because one person insisted on putting 100,000 AD. The median might be a better measure in this case; it was mid-2067.

Thomas Edison patented the lightbulb in 1880. I've never before been a firm believer in the wisdom of crowds, but it really came through in this case. Even though this was clearly not an easy question and many people got really far-off answers, the mean was 1879.3 and the median was 1880. The standard deviation was 36.1. Person who put "2172", you probably thought you were screwing up the results, but in fact you managed to counterbalance the other person who put "1700", allowing the mean to revert back to within one year of the correct value :P

The average person was 26.77% sure they got within 5 years of the correct answer on the lightbulb question. 30% of people did get within 5 years. I'm not sure how much to trust the result, because several people put the exact correct year down and gave it 100% confidence. Either they were really paying attention in history class, or they checked Wikipedia. There was a high correlation between high levels of confidence on the question and actually getting the question right, significant at the <.001 level.

I ran some correlations between different things, but they're nothing very interesting. I'm listing the ones that are significant at the <.05 level, but keep in mind that since I just tried correlating everything with everything else, there are a couple hundred correlations and it's absolutely plausible that many things would achieve that significance level by pure chance.

How long you've been in the community obviously correlates very closely with how much of Robin and Eliezer's posts you've read (and both correlate with each other).

People who have read more of Robin and Eliezer's posts have higher karma. People who spend more time per day on Less Wrong have higher karma (with very strong significance, at the <.001 level.)

People who have been in the community a long time and read many of EY and RH's posts are more likely to believe in Many Worlds and Cryonics, two unusual topics that were addressed particularly well on Overcoming Bias. That suggests if you're a new person who doesn't currently believe in those two ideas, and they're important to you, you might want to go back and find the OB sequences about them (here's Many Worlds, and here's some cryonics). There were no similar effects on things like belief in God or belief in aliens.

Older people were less likely to spend a lot of time on the site, less likely to believe in Many Worlds, less likely to believe in global warming, and more likely to believe in aliens.

Everything in the God/revealed religion/supernatural cluster correlated pretty well with each other. Belief in cryonics correlated pretty well with belief in anti-agathics.

Here is an anomalous finding I didn't expect: the higher a probability you assign to the truth of revealed religion, the less confident you are that your IQ is above average (even though no correlation between this religious belief and IQ was actually found). Significance is at the .025 level. I have two theories on this: first, that we've been telling religious people they're stupid for so long that it's finally starting to sink in :) Second, that most people here are not religious, and so the people who put a "high" probability for revealed religion may be assigning it 5% or 10%, not because they believe it but because they're just underconfident people who maybe overadjust for their biases a little much. This same underconfidence leads them to underestimate the possibility that their IQ is above average.

The higher probability you assign to the existence of aliens in the universe, the more likely you are to think we'll survive until 2100 (p=.002). There is no similar correlation for aliens in the galaxy. I credit the Great Filter article for this one too - if no other species exist, it could mean something killed them off.

And, uh, the higher probability you assign to the existence of aliens in the galaxy (but not in the universe) the more likely you are (at a .05 sig) to think global warming is man-made. I have no explanation for this one. Probably one of those coincidences.

Moving on - of the 102 people who cared about the ending to 3 Worlds Collide, 68 (66.6%) prefered to see the humans blow up Huygens, while 34 (33.3%) thought we'd be better off cooperating with the aliens and eating delicious babies.

Of the 114 people who had opinions about the Singularity, 85 (74.6%) go with Eliezer's version, and 29 (25.4%) go with Robin's.

If you're playing another Less Wronger in the Prisoner's Dilemma, you should know that of the 133 who provided valid information for this question, 96 (72.2%) would cooperate and 37 (27.8%) would defect. The numbers switch when one player becomes an evil paper-clip loving robot; out of 126 willing to play the "true" Prisoner's Dilemma, only 42% cooperate and 58% defect.

Of the 124 of us willing to play the Counterfactual Mugging, 53 (42.7%) would give Omega the money, and 71 (57.3%) would laugh in his face.

Of the 146 of us who had an opinion on aid to Africa, 24 (16.4%) thought it was almost always a good thing, 42 (27.8%) thought it was almost always a bad thing, and 80 (54.8%) took a middle-of-the-road approach and said it could be good, but only in a few cases where it was done right.

Of 128 of us who wanted to talk about our moral theories, 94 (73.4%) were consequentialists, about evenly split between garden-variety or Eliezer-variety (many complained they didn't know what Eliezer's interpretation was, or what the generic interpretation was, or that all they knew was that they were consequentialists). 15 (9%) said with more or fewer disclaimers that they were basically deontologists, and 5 (3.9%) wrote-in virtue ethics, and objected to their beliefs being left out (sorry!). 14 people (10.9%) didn't believe in morality.

Despite the seemingly overwhelming support for cryonics any time someone mentions it, only three of us are actually signed up! Of the 161 of us who admitted we weren't, 11 (6.8%) just never thought about it, 99 (59.6%) are still considering it, and 51 (31.7%) have decided against it.

Beware Trivial Inconveniences

90 Post author: Yvain 06 May 2009 10:04PM

The Great Firewall of China. A massive system of centralized censorship purging the Chinese version of the Internet of all potentially subversive content. Generally agreed to be a great technical achievement and political success even by the vast majority of people who find it morally abhorrent.

I spent a few days in China. I got around it at the Internet cafe by using a free online proxy. Actual Chinese people have dozens of ways of getting around it with a minimum of technical knowledge or just the ability to read some instructions.

The Chinese government isn't losing any sleep over this (although they also don't lose any sleep over murdering political dissidents, so maybe they're just very sound sleepers). Their theory is that by making it a little inconvenient and time-consuming to view subversive sites, they will discourage casual exploration. No one will bother to circumvent it unless they already seriously distrust the Chinese government and are specifically looking for foreign websites, and these people probably know what the foreign websites are going to say anyway.

Think about this for a second. The human longing for freedom of information is a terrible and wonderful thing. It delineates a pivotal difference between mental emancipation and slavery. It has launched protests, rebellions, and revolutions. Thousands have devoted their lives to it, thousands of others have even died for it. And it can be stopped dead in its tracks by requiring people to search for "how to set up proxy" before viewing their anti-government website.

I was reminded of this recently by Eliezer's Less Wrong Progress Report. He mentioned how surprised he was that so many people were posting so much stuff on Less Wrong, when very few people had ever taken advantage of Overcoming Bias' policy of accepting contributions if you emailed them to a moderator and the moderator approved. Apparently all us folk brimming with ideas for posts didn't want to deal with the aggravation.

Okay, in my case at least it was a bit more than that. There's a sense of going out on a limb and drawing attention to yourself, of arrogantly claiming some sort of equivalence to Robin Hanson and Eliezer Yudkowsky. But it's still interesting that this potential embarrassment and awkwardness was enough to keep the several dozen people who have blogged on here so far from sending that "I have something I'd like to post..." email.

Companies frequently offer "free rebates". For example, an $800 television with a $200 rebate. There are a few reasons companies like rebates, but one is that you'll be attracted to the television because it appears to have a net cost only $600, but then filling out the paperwork to get the rebate is too inconvenient and you won't get around to it. This is basically a free $200 for filling out an annoying form, but companies can predict that customers will continually fail to complete it. This might make some sense if you're a high-powered lawyer or someone else whose time is extremely valuable, but most of us have absolutely no excuse.

One last example: It's become a truism that people spend more when they use credit cards than when they use money. This particular truism happens to be true: in a study by Prelec and Simester1, auction participants bid twice as much for the same prize when using credit than when using cash. The trivial step of getting the money and handing it over has a major inhibitory effect on your spending habits.

I don't know of any unifying psychological theory that explains our problem with trivial inconveniences. It seems to have something to do with loss aversion, and with the brain's general use of emotion-based hacks instead of serious cost-benefit analysis. It might be linked to akrasia; for example, you might not have enough willpower to go ahead with the unpleasant action of filling in a rebate form, and your brain may assign it low priority because it's hard to imagine the connection between the action and the reward.

But these trivial inconveniences have major policy implications. Countries like China that want to oppress their citizens are already using "soft" oppression to make it annoyingly difficult to access subversive information. But there are also benefits for governments that want to help their citizens.

"Soft paternalism" means a lot of things to a lot of different people. But one of the most interesting versions is the idea of "opt-out" government policies. For example, it would be nice if everyone put money into a pension scheme. Left to their own devices, many ignorant or lazy people might never get around to starting a pension, and in order to prevent these people's financial ruin, there is strong a moral argument for a government-mandated pension scheme. But there's also a strong libertarian argument against that idea; if someone for reasons of their own doesn't want a pension, or wants a different kind of pension, their status as a free citizen should give them that right.

The "soft paternalist" solution is to have a government-mandated pension scheme, but allow individuals to opt-out of it after signing the appropriate amount of paperwork. Most people, the theory goes, would remain in the pension scheme, because they understand they're better off with a pension and it was only laziness that prevented them from getting one before. And anyone who actually goes through the trouble of opting out of the government scheme would either be the sort of intelligent person who has a good reason not to want a pension, or else deserve what they get2.

This also reminds me of Robin's IQ-gated, test-requiring would-have-been-banned store, which would discourage people from certain drugs without making it impossible for the true believers to get their hands on them. I suggest such a store be located way on the outskirts of town accessible only by a potholed road with a single traffic light that changes once per presidential administration, have a surly clerk who speaks heavily accented English, and be open between the hours of two and four on weekdays.

 

Footnotes

1: See Jonah Lehrer's book How We Decide. In fact, do this anyway. It's very good.

2: Note also the clever use of the status quo bias here.

In response to Step Back
Comment author: Yvain 09 May 2009 10:34:35PM 3 points [-]

You may also want to try moving your eyes back and forth.

Comment author: mattnewport 06 May 2009 10:35:57PM 0 points [-]

If I want to divide up X units of torture between two animals, one of which is cuter than the other, from a purely consequentialist position there's no reason to prefer one to the other.

Well, humans seem to be more upset by images of baby seals being clubbed than by the death of less cute but similarly 'conscious' creatures so that might factor into your total suffering calculation but that aside this does seem to follow from your premises.

It might help if you think of me as trying to minimize the number of suffering*consciousness units.

Why is that preference uniquely privileged though? What justifies it over preferring to minimize the number of suffering*(value I assign to animal) units? If I value something about dogs over pigs (lets call it 'empathy units' because that is something like a description of the source of my preference) why is that a less justified choice of preference than 'consciousness'?

If you just genuinely value what you're calling 'consciousness' here over any other measure of value that's a perfectly reasonable position to take. You seem to want to universalize the preference though and I get the impression that you recognize that it goes against most people's instinctive preferences. If you want to persuade others to accept your preference ranking (maybe you don't - it's not clear to me) then I think you need to come up with a better justification. You should also bear in mind you may find yourself arguing to sacrifice humanity for a super-conscious paperclip maximizer - is that really a position you want to take?

Comment author: Yvain 06 May 2009 11:02:38PM 3 points [-]

Well, I admit to being one of the approximately seven billion humans who can't prove their utility functions from first principles. But I think there's a very convincing argument that consciousness is in fact what we're actually looking for and naturally taking into account.

Happiness only is happiness, and pain only is pain, insofar as it is perceived by awareness. If a scientist took a nerve cell with a pain receptor, put it in a Petri dish, and stimulated it for a while, I wouldn't consider this a morally evil act.

I find in my own life that different levels of awareness correspond to different levels of suffering. Although something bad happening to me in a dream is bad, I don't worry about it nearly as much as I would if it happened when I was awake and fully aware. Likewise, if I'm zonked out on sedatives, I tend to pay less attention to my own pain.

I hypothesize that different animals have different levels of awareness, based on intuition and my knowledge of their nervous systems. In this case, they would be able to experience different levels of suffering. What I meant by saying my utility function multiplied suffering by awareness would have been better phrased as:

Suffering = bad things*awareness

while trying to minimize suffering. This is why, for example, doing all sorts of horrible things to a rock is a morally neutral act, doing them to an insect is probably bad but not anything to lose sleep over, and doing them to a human is a moral problem even if it's a human I don't personally like.

Your paperclip example is a classical problem called the utility monster. I don't really have any especially brilliant solution beyond what has already been said about the issue. To some degree I bite the bullet: if there was some entity whose nervous system was so acute that causing it the slightest amount of pain would correspond to 3^^^3 years of torture for a human being, I'd place high priority on keeping that entity happy.

Comment author: Eliezer_Yudkowsky 06 May 2009 10:35:01PM 0 points [-]

There are two of you?

Comment author: Yvain 06 May 2009 10:41:20PM *  17 points [-]

"Soft paternalism relies for its justification on the notion that each of us contains multiple selves"

-- New York Times, The New, Soft Paternalism

Comment author: MBlume 06 May 2009 10:11:56PM 19 points [-]

Most people, the theory goes, would remain in the pension scheme, because they understand they're better off with a pension and it was only laziness that prevented them from getting one before.

I've felt for a long time that the same solution should be implemented for organ donation.

(Actually, there's a case to be made for "screw your sentimental attachment to your meat parts -- we can save lives". But soft paternalism is a start.)

Comment author: Yvain 06 May 2009 10:19:46PM *  10 points [-]

I agree with you 200%. I think a couple of countries in Europe might have that. I heard Brazil used to have it, but had to change it when stupid people got angry.

Comment author: mattnewport 06 May 2009 09:33:43PM 1 point [-]

Any personal preference I have for dogs is because they're cuter than pigs are, which seems like a bad way to make moral decisions.

I think you've deliberately muddied the waters by throwing in the word 'cute' there. You justify your general rule for preferring some lifeforms to others by saying you value 'consciousness' but then say that preferring dogs over pigs for 'cuteness' is not a good way to make moral decisions. If you take away the loaded words all you're really saying in both cases is that you value animal A more than animal B because it has more of property X. When X is consciousness that's a good justification, when it's cuteness it's a bad justification.

I'm quite happy to just say that I prefer some animals to others and I value them accordingly. That preference is a combination of factors which I couldn't give you a formula for but I don't feel I need to do so to justify following my preference. In the case of dogs I think it's more than cuteness - they are pack hunting animals that have been bred over many generations to live with humans as companions (rather than as livestock) and so it is not unsurprising that we should have affinity for them. Preferring them over pigs seems no more problematic than preferring a friendly AI over a paperclip maximizer - they share more common goals with us than pigs do.

Given the low probability of me actually going through with donating $1 more to charity just because I bought a $1 cheaper chicken, I'd probably take the more expensive one, though.

That's not a very rational approach. If it's easier, think of it as $150 a year (probably ballpark for me based on my own chicken consumption) and consider what charity you could donate $150 extra to. In my opinion being rational about personal finances is a pretty good starting place for an aspiring rationalist.

Comment author: Yvain 06 May 2009 10:15:25PM *  3 points [-]

I don't interpret "consciousness" as a preference giving some animals more value to me than others. I interpret it as a multiplier that needs to be used in order to even out preferences.

Let's say I want to minimize suffering in a target-independent way, but I need to divide X units of torture between a human and an ant. I would choose to apply all X units to the ant, not just because I like humans more than ants, but because that decision actually minimizes total suffering. My wild guess is that ants can't really suffer all that much; they probably get some vague negative feeling but it's (again, I am guessing wildly) nothing like as strong or as painful as the pain that a human, with their million times more neurons, feels.

In contrast, obviously cuteness has no effect on level of suffering. If I want to divide up X units of torture between two animals, one of which is cuter than the other, from a purely consequentialist position there's no reason to prefer one to the other.

It might help if you think of me as trying to minimize the number of suffering*consciousness units. That's why I wouldn't care about eating TAW's genetically engineered neuronless cow, and it's why I care less about ants than humans.

(or a metaphor: let's say a hospital administrator has to distribute X organs among needy transplant patients. Even if the hospital administrator chooses to be unbiased regarding the patients' social value - ie not prefer a millionaire to a bum - the administrator still has a good case for giving the organ to someone for whom it will bring them 50 more years of life rather than 6 more months. That's a completely different kind of preference than 'I like this guy better'. The administrator is trying to impartially maximize lives saved*years)

Hopefully that makes it clear what the difference between this theory and "preferring" cute animals is.

Beware Trivial Inconveniences

90 Post author: Yvain 06 May 2009 10:04PM

The Great Firewall of China. A massive system of centralized censorship purging the Chinese version of the Internet of all potentially subversive content. Generally agreed to be a great technical achievement and political success even by the vast majority of people who find it morally abhorrent.

I spent a few days in China. I got around it at the Internet cafe by using a free online proxy. Actual Chinese people have dozens of ways of getting around it with a minimum of technical knowledge or just the ability to read some instructions.

The Chinese government isn't losing any sleep over this (although they also don't lose any sleep over murdering political dissidents, so maybe they're just very sound sleepers). Their theory is that by making it a little inconvenient and time-consuming to view subversive sites, they will discourage casual exploration. No one will bother to circumvent it unless they already seriously distrust the Chinese government and are specifically looking for foreign websites, and these people probably know what the foreign websites are going to say anyway.

Think about this for a second. The human longing for freedom of information is a terrible and wonderful thing. It delineates a pivotal difference between mental emancipation and slavery. It has launched protests, rebellions, and revolutions. Thousands have devoted their lives to it, thousands of others have even died for it. And it can be stopped dead in its tracks by requiring people to search for "how to set up proxy" before viewing their anti-government website.

I was reminded of this recently by Eliezer's Less Wrong Progress Report. He mentioned how surprised he was that so many people were posting so much stuff on Less Wrong, when very few people had ever taken advantage of Overcoming Bias' policy of accepting contributions if you emailed them to a moderator and the moderator approved. Apparently all us folk brimming with ideas for posts didn't want to deal with the aggravation.

Okay, in my case at least it was a bit more than that. There's a sense of going out on a limb and drawing attention to yourself, of arrogantly claiming some sort of equivalence to Robin Hanson and Eliezer Yudkowsky. But it's still interesting that this potential embarrassment and awkwardness was enough to keep the several dozen people who have blogged on here so far from sending that "I have something I'd like to post..." email.

Companies frequently offer "free rebates". For example, an $800 television with a $200 rebate. There are a few reasons companies like rebates, but one is that you'll be attracted to the television because it appears to have a net cost only $600, but then filling out the paperwork to get the rebate is too inconvenient and you won't get around to it. This is basically a free $200 for filling out an annoying form, but companies can predict that customers will continually fail to complete it. This might make some sense if you're a high-powered lawyer or someone else whose time is extremely valuable, but most of us have absolutely no excuse.

One last example: It's become a truism that people spend more when they use credit cards than when they use money. This particular truism happens to be true: in a study by Prelec and Simester1, auction participants bid twice as much for the same prize when using credit than when using cash. The trivial step of getting the money and handing it over has a major inhibitory effect on your spending habits.

I don't know of any unifying psychological theory that explains our problem with trivial inconveniences. It seems to have something to do with loss aversion, and with the brain's general use of emotion-based hacks instead of serious cost-benefit analysis. It might be linked to akrasia; for example, you might not have enough willpower to go ahead with the unpleasant action of filling in a rebate form, and your brain may assign it low priority because it's hard to imagine the connection between the action and the reward.

But these trivial inconveniences have major policy implications. Countries like China that want to oppress their citizens are already using "soft" oppression to make it annoyingly difficult to access subversive information. But there are also benefits for governments that want to help their citizens.

"Soft paternalism" means a lot of things to a lot of different people. But one of the most interesting versions is the idea of "opt-out" government policies. For example, it would be nice if everyone put money into a pension scheme. Left to their own devices, many ignorant or lazy people might never get around to starting a pension, and in order to prevent these people's financial ruin, there is strong a moral argument for a government-mandated pension scheme. But there's also a strong libertarian argument against that idea; if someone for reasons of their own doesn't want a pension, or wants a different kind of pension, their status as a free citizen should give them that right.

The "soft paternalist" solution is to have a government-mandated pension scheme, but allow individuals to opt-out of it after signing the appropriate amount of paperwork. Most people, the theory goes, would remain in the pension scheme, because they understand they're better off with a pension and it was only laziness that prevented them from getting one before. And anyone who actually goes through the trouble of opting out of the government scheme would either be the sort of intelligent person who has a good reason not to want a pension, or else deserve what they get2.

This also reminds me of Robin's IQ-gated, test-requiring would-have-been-banned store, which would discourage people from certain drugs without making it impossible for the true believers to get their hands on them. I suggest such a store be located way on the outskirts of town accessible only by a potholed road with a single traffic light that changes once per presidential administration, have a surly clerk who speaks heavily accented English, and be open between the hours of two and four on weekdays.

 

Footnotes

1: See Jonah Lehrer's book How We Decide. In fact, do this anyway. It's very good.

2: Note also the clever use of the status quo bias here.

Comment author: Yvain 05 May 2009 11:27:45PM *  12 points [-]

Color me skeptical. Consider that if David's winning percentage is 63.6, being weaker is actually an advantage, and Goliath's optimal strategy is to abandon most of your resources in order to become the new weakling. That doesn't make a lot of sense.

I'd really like to see Toft's criteria for including wars, for deciding the winner, and for deciding whether David was using the optimal strategy. The only unbiased way to do this would be to have the categorization done by a third party who had no idea what the object of the experiment was. For best results, by several people working independently. And did he include all the hundreds of little wars between the US and various Native American tribes?

Or did he just take some famous wars (probably famous precisely because they were very close or had surprising results), call the ones where the smaller side did really well "David using an unconventional strategy", and then proclaim an unusual number of victories?

Even if the experiment was done correctly, asymmetric conflicts in wars can't really be generalized. Often it's something like "Big country tries to colonize little country", in which case little country is fighting on its own ground and just needs to make big country annoyed enough to go away. There's a big difference between that and basketball.

Speaking of basketball, in a lot of games there are successful but unstable strategies. Imagine a game where there are strategies 1-99, where each strategy is more likely to win, but also takes an increasing level of expertise. Imagine also that Strategy 5 has a special property - it beats any strategy numbered in the 90s, but no other. Imagine also that it takes some training investment to use a strategy - you can't just switch strategy numbers willy-nilly.

Matches between masters of this game will tend to look like 98 vs. 95 or something. A beginner may step in, use Strategy 5, and win a few games against masters who were expecting him to also be a master. But eventually the masters will switch to Strategy 89, and massacre the beginner. After all annoying beginners have been eliminated, the masters will go back to fighting among themselves at the 90s level. I can't think of a good example now, but I remember exactly this sort of thing happening in a few sports.

I wonder if the full-court press might be equivalent to strategy 5 here - something that other basketball teams could defeat, if they put in a bit of extra training, but since everyone knows that no one uses it no one trains to defeat it.

One last thing: I think my post Help, Help, I'm Being Oppressed is relevant here. The article plays a kind of dirty trick to get our sympathy on David's side: it says that the Goliaths are trying to alter the rules to form a cartel to prevent David's brilliant but unconventional strategies from allowing him to compete on a level playing field. Because that's how overprotective that evil Goliath is of his advantages and power.

But imagine the situation was reversed. David, through sheer pluck and spunk, was managing to win in the traditional way, and Goliath used a dirty trick generally considered to be against the rules, like using his vast monetary resources to hire a computer scientist to design an unbeatable fleet by AI. We would immediately scream "foul!" and be outraged that David's shot at the prize was being taken away so unfairly.

So we need to make sure we're uncoupling the question of whether we're rooting for David or Goliath, from the question of whether it's okay to win using dirty tricks that violate the spirit of a game and potentially make it less fun for everyone else.

Comment author: taw 04 May 2009 07:48:03PM 4 points [-]

I don't see this as a value-disagreement case. Someone who has different values, and behaves in a way that's broadly consistent with these values, is on the consequentialist side. People who just follow certain rituals (like not eating meat), and claim to have some values but don't act in a way consistent with them, are on non-consequentialist side.

I've never seen anybody who was vegetarian because of value disagreements, and was behaving consistently with their alleged values.

For example if you claim to prefer non-existence of animals to them being used as food, then you clearly must support destruction of all nature reserves, as that's exactly the same choice. And if you're against animal suffering, you'd be totally happy to eat cows genetically modified not to have pain receptors. And so on. All positions never taken by any vegetarians.

Comment author: Yvain 05 May 2009 10:56:01PM 4 points [-]

For example if you claim to prefer non-existence of animals to them being used as food, then you clearly must support destruction of all nature reserves, as that's exactly the same choice. And if you're against animal suffering, you'd be totally happy to eat cows genetically modified not to have pain receptors. And so on. All positions never taken by any vegetarians.

I don't agree with the premise of the first position, but I agree wholeheartedly with the second (well, replacing "pain receptors" with a complete rework of the mammalian brain and nervous system, since just removing pain receptors is a very limited kind of alleviation of suffering. After all, I could remove your literal pain receptors and lock you in a 6x6 cell for your whole life, and you'd still be suffering.)

I hope now you'll never again have to say it's a position never taken by any vegetarian.

In response to Bead Jar Guesses
Comment author: Yvain 04 May 2009 09:09:49PM 0 points [-]

Do you see this as being sort of like Jimmy's metauncertainty?

Also, if Omega pulled out a bead and then asked you about the next one, the Rule of Succession would be a good place to start making guesses.

Comment author: MorganHouse 04 May 2009 12:05:41AM 0 points [-]

Kin selection suggests that chickens may care about their siblings, and general evolution suggests they definitely care about their children.

What I meant is that birds' programming doesn't feature advanced mental concepts like "care", but simple instinctive responses (that can be easily triggered with false stimuli) take their place. However, I see now that this was not important to my point, and I could have left it out, in place of "don't care whether other species live or die".

If you set some criteria for "deserving" our respect, of course a lot of animals can't live up to it. But it doesn't seem right to use that as justification for hurting them.

What's so inherently bad about pain? Is it morally questionable to run a piece of control software for a cleaning robot, that has a "const bool in_pain = true;"?

Now he can't care about other people, but his thought and emotions are otherwise intact. Is it now okay to torture Bob?

With his intelligence intact, he can still be valuable to us, and depending on what he did in the past, we may be in moral debt to him. However, if he was born with no mental facilities outside of those of a chicken, my foremost reason for keeping him alive would be to prevent an emotional impact for other people.

Comment author: Yvain 04 May 2009 12:16:14AM 7 points [-]

What's so inherently bad about pain?

The proper way to prove that pain is bad is proof by induction: specifically, hook an electric wire to the testicles of the person who doesn't think pain is bad, induce a current, and continue it until the person admits that pain is bad (this is also the proper way to prove that creationism is false, or at least the most fun).

Is it morally questionable to run a piece of control software for a cleaning robot, that has a "const bool in_pain = true;"?

This is getting into the subject of qualia, which I freely admit to not understanding. But I'm pretty sure I have some, and I'm pretty sure they're harder to produce than a variable with the label "pain".

With his intelligence intact, he can still be valuable to us, and depending on what he did in the past, we may be in moral debt to him.

I'd guess from this statement that you're either not a consequentialist, or you're some exotic type of consequentialist straight out of Alicorn's syllabus. If you clarify exactly what your moral theory is, I can give you a better estimate on how likely we are to be talking past each other because we have completely different premises.

Comment author: MorganHouse 03 May 2009 10:39:32PM *  9 points [-]

Ethical. If I wouldn't want people torturing dogs, I have no justification to be okay with people torturing cows, pigs, and chickens

Dogs are genetically selected for living together with humans. As such, and unlike their wolf predecessors, dogs are friendly towards us. In many cases, care is reciprocal, in that we more often care about people who care about us. I propose that chickens don't have even the slightest sense of morality, and don't care whether their siblings live or die. With this in mind, I think it's a somewhat justified to torture birds and low mammals, since they don't care about our or their families' well-being to begin with.

However, I would never torture a chicken unless I was at least 99% sure it had valuable information, and the future of the farm was at stake.

Comment author: Yvain 03 May 2009 11:06:49PM 8 points [-]

Kin selection suggests that chickens may care about their siblings, and general evolution suggests they definitely care about their children.

...which is exactly the problem. You sound like you're holding a grudge against chickens for not being evolutionarily programmed in a certain way. Let it go. If you set some criteria for "deserving" our respect, of course a lot of animals can't live up to it. But it doesn't seem right to use that as justification for hurting them.

Thought experiment: I take Bob and cut out the part of his brain involved in empathy. Now he can't care about other people, but his thought and emotions are otherwise intact. Is it now okay to torture Bob?

Comment author: mattnewport 03 May 2009 09:35:08PM 2 points [-]

If I wouldn't want people torturing dogs, I have no justification to be okay with people torturing cows, pigs, and chickens, and from what I've seen conditions in a lot of farms and slaughterhouses are tantamount to torture.

Do you place equal value on the wellbeing of all animals? This sounds like the same kind of dogmatic adherence to equal weighting that I have a problem with in utilitarianism. I don't want people torturing dogs, I'm less concerned about people torturing chickens. I value the wellbeing of dogs more than the well-being of chickens. I value both considerably less than the wellbeing of humans and considerably more than the wellbeing of HIV viruses.

All else being equal, I'd prefer less rather than more chicken-suffering. If however I have a choice between a $5 chicken breast that caused X chicken-suffering and a $6 chicken breast that caused 0.5X chicken-suffering I'll save the extra dollar and apply it to something I consider more important than chicken-suffering. A donation to a puppy rescue shelter for example (though that would be low on my overall list of priorities).

Comment author: Yvain 03 May 2009 11:00:05PM 10 points [-]

I weight the well-being of animals in proportion to what I would call for lack of a better word their consciousness. I think dolphins are probably self-aware, capable of reflection, and have strong senses of pain and pleasure. I think ants are probably much less so, although still nonzero. So I place much less emphasis upon the well-being of ants than upon the well-being of dolphins. Since viruses have no nervous system and no brain, I'm prepared to give them zero value.

However, I have no evidence that dogs are more aware than pigs are. Any personal preference I have for dogs is because they're cuter than pigs are, which seems like a bad way to make moral decisions. So I am not prepared to make pigs less valuable than dogs.

I never thought about it in terms of your two-different-kinds-of-chicken-breast problem, but I would agree that this would require an actual calculation to see whether the money saved could prevent more suffering than was caused to the chicken. Given the low probability of me actually going through with donating $1 more to charity just because I bought a $1 cheaper chicken, I'd probably take the more expensive one, though.

Comment author: badger 03 May 2009 07:10:42PM 5 points [-]

Inspired by mattnewport's comment, here are additional questions for any vegetarians: if vat-grown meat were developed, would you eat it? Would there be any ethical issues with eating it?

Comment author: Yvain 03 May 2009 09:15:15PM 4 points [-]

I would happily eat vat meat without a single pang of conscience.

Possible exception: if it was getting to the point where farm animals were becoming endangered, I would expect a movement supporting traditional farms to arise, and for this movement to place a high priority on animal welfare. If this happened, I would support this movement by buying farm-grown meat, but this would be a personal preference and I would not recognize a moral obligation to do so.

Comment author: Yvain 03 May 2009 09:12:49PM 6 points [-]
  1. I don't eat meat.
  2. Ethical. If I wouldn't want people torturing dogs, I have no justification to be okay with people torturing cows, pigs, and chickens, and from what I've seen conditions in a lot of farms and slaughterhouses are tantamount to torture. Even though animals can't think verbally, they still have some level of awareness and the ability to feel pain, so causing them suffering is verboten. I am kind of sympathetic to the argument that free range meat raised with the animals' welfare in mind isn't so bad, and to the argument that if we weren't raising these animals for food they'd probably be endangered or extinct. But free range is only a small percent of meat products, and there are major environmental costs anyway, and the meat-farming industry just does so much damage in so many ways that I feel I need to do my part to discourage it. Right now my goal is to aim for zero meat and accept the inevitable lapses when they come as not being an ethical disaster.
  3. I'm not too strict about it. When I'm traveling or a guest somewhere it's pretty tough to avoid meat, so I let myself get away with it.
  4. Hard to tell. I think I'd at least share my reasons with them, but if they didn't want to that's their choice. As long as they can provide a rational explanation, of course :)
  5. Never tried.
  6. I eat a lot of Quorn when I'm in the British Isles, and soy products when I'm elsewhere. Quorn is better, but I haven't been able to find it outside Britain and Ireland.
  7. I'm pretty live-and-let-live about this.
  8. Became a vegetarian in elementary school, I think, maybe middle school. Gave it up on three or four occasions for a few months, usually after moving and not being able to find good vegetarian foods there, but always went back. Sometimes give it up for a few months when I go back to my parents' place, because the food there is too good and I don't have as much control over my diet.
  9. I love meat and I want it all the time.
  10. I don't really eat many fruits or vegetables. I hate them to the point where I have trouble keeping them down. This doesn't apply as much to salads. So I kind of live off of grain products, with some milk and eggs and Quorn thrown in. There are a lot of diet theories that suggest I should be very fat right now, but I'm actually pretty thin. Go figure.
In response to Return of the Survey
Comment author: conchis 03 May 2009 12:25:15PM 1 point [-]

I assume that in the political views question, under the liberal option, it should say "UK Labour Party" rather than "UK Tories"

Comment author: Yvain 03 May 2009 12:41:40PM 0 points [-]

Argh, you're right. Changed, along with one or two of kpreid's points.

In response to Return of the Survey
Comment author: AllanCrossman 03 May 2009 09:59:02AM *  1 point [-]

There are little checkmarks under the headings for section 2 and 3 - do those mean anything?

Comment author: Yvain 03 May 2009 11:45:04AM 0 points [-]

They mean I'm very bad at writing surveys, and I couldn't figure out how to make a section divider without telling the code it was a "question". All "questions" have to have at least one answer, which in the case of the section dividers was that checkmark. Anyone familiar with Google Docs and know how I can avoid that next time?

Comment author: trimeta 03 May 2009 07:12:50AM 2 points [-]

While I'm sympathetic to the idea that "you are likely to begin believing things you say to yourself frequently, and furthermore you are likely to act in ways to make these beliefs come true," I'm not sure I'd use Scott Adams as the prototypical purveyor of this idea. He is known to have beliefs difficult to characterize as "rational," notably a belief in intelligent design creationism. The Wikipedia also cites him as being a hypnotist and a vegan; these are not inherently irrational, but I don't have further sources to analyze Adam's specific perspective on these issues.

Comment author: Yvain 03 May 2009 11:40:59AM *  5 points [-]

The intelligent design issue is complex, but he's said outright that he doesn't believe in it. I think his position is something like "Most people who believe evolution are not smart enough to understand it, and would be better off believing intelligent design since it makes more sense on a naive level. Most believers in evolution who are not biologists are making the 'science as belief-attire' type mistake." It's been a while since I read about that particular flame war, so I might be mistaken, but I do remember he specifically said that with extremely high probability ID was wrong.

Hypnotism has been shown to work in studies by the AMA, BMJ, and every other group of medical experts who have investigated the question, and he's a vegetarian, not a vegan - and so am I, so you're going to have trouble convincing me that's a strike against him. Though if you want to write a post about it, I'd be interested in hearing your arguments against.

In response to Return of the Survey
Comment author: Alicorn 03 May 2009 02:59:03AM *  6 points [-]

<karmawhore>I took the survey!</karmawhore>

Edit: Somebody took my shiny new karma point. That was not very nice of them, since it was explicitly offered as reciprocation for survey participation.

Edit 2: It is back, or has been replaced by someone else. Thanks!

Edit 3: ...I'm off to cognitive-dissonance myself into believing that I just like taking surveys.

Comment author: Yvain 03 May 2009 03:04:15AM *  1 point [-]

Yay!
I just sent you a private message; please check your inbox.

Return of the Survey

13 Yvain 03 May 2009 02:10AM

[UPDATE: Survey is now closed. Thanks to everyone who took it. Results soon. Ignore everything below.]

Last week, I asked people for help writing a survey. I've since taken some of your suggestions. Not all, because I wanted to keep the survey short, and because the survey software I'm using made certain types of questions inconvenient, but some. I hope no one's too angry about their contributions being left out.

Please note that, due to what was very possibly a bad decision on my part as to what would be most intuitive, I've requested all probabilities be in percentage format. So if you think something has a 1/2 chance of being true, please list 50 instead of .5.

Please take the survey now; it can be found here and it shouldn't take more than fifteen or twenty minutes. Unless perhaps you need to spend a lot of time determining your opinions on controversial issues, in which case it will be time well spent!

Several people, despite the BOLD ALL CAPS TEXT saying not to take the survey in the last post, went ahead and took the survey. Your results have been deleted. Please take it again. Thank you.

I'll leave this open for about a week, calculate some results, then send out the data. There is an option to make your data private at the bottom of the survey.

Thanks to everyone who takes this. If you want, post a comment saying you took it below, and I'll give you a karma point :)

What I Tell You Three Times Is True

46 Post author: Yvain 02 May 2009 11:47PM

"The human brain evidently operates on some variation of the famous principle enunciated in 'The Hunting of the Snark': 'What I tell you three times is true.'"

   -- Norbert Weiner, from Cybernetics

Ask for a high-profile rationalist, and you'll hear about Richard Dawkins or James Randi or maybe Peter Thiel. Not a lot of people would immediately name Scott Adams, creator of Dilbert. But as readers of his blog know, he's got a deep interest in rationality, and sometimes it shows up in his comics: for example, this one from last week. How many people can expose several million people to the phrase "Boltzmann brain hypothesis" and have them enjoy it?

So I was very surprised to find Adams was a believer in and evangelist of something that sounded a lot like pseudoscience. "Affirmations" are positive statements made with the belief that saying the statement loud enough and long enough will help it come true. For example, you might say "I will become a syndicated cartoonist" fifteen times before bed every night, thinking that this will in fact make you a syndicated cartoonist. Adams partially credits his success as a cartoonist to doing exactly this.

He admits "it sounds as if I believe in some sort of voodoo or magic", and acknowledges that "skeptics have suggested, and reasonably so, that this is a classic case of selective memory" but still swears that it works. He also has "received thousands of e-mails from people recounting their own experiences with affirmations. Most people seem to be amazed at how well they worked."

None of this should be taken too seriously without a controlled scientific study investigating it, of course. But is it worth the effort of a study, or should it be filed under "so stupid that it's not worth anyone's time to investigate further"?

I think there's a good case to be made from within a rationalist/scientific worldview that affirmations may in fact be effective for certain goals. Not miraculously effective, but not totally useless either.

To build this case, I want to provide evidence for two propositions. First, that whether we subconsciously believe we can succeed affects whether or not we succeed. Second, that repeating a statement verbally can make the subconscious believe it.

The link between belief in success and success has progressed beyond the motivational speaker stage and into the scientific evidence stage. The best-known of these links is the placebo effect. For certain diseases, believing that you will get better does increase your probability of getting better. This works not only subjectively (ie you feel less pain) but objectively (ie ulcers heal more quickly, inflammation decreases).

The placebo effect applies in some stranger cases outside simple curative drugs. A placebo stop-smoking pill does increase your chance of successfully quitting tobacco. Placebo strength pills enable you to run faster and lift more weight. Placebo alcohol makes you more gregarious and less inhibited1. Placebo therapies for phobia can desensitize you to otherwise terrifying stimuli.

There are some great studies about the effect of belief in school settings. Pick a student at random and tell the teacher that she's especially smart, and by the end of the year she will be doing exceptionally well; tell the teacher that she is exceptionally stupid, and by the end of the year she'll be doing exceptionally poorly. The experimenters theorized that the teacher's belief about the student's intelligence was subconsciously detected by the student, and that the student was somehow adjusted her performance to fit that belief. In a similar study, minority students were found to do worse on tests when reminded of stereotypes that minorities are stupid, and better when tested in contexts that downplayed their minority status, suggesting that the students' belief that they would fail was enough to make them fail.

Belief can also translate to success when mediated by signals of dominance and confidence. We've already discussed how hard-to-fake signals of confidence can help someone pick up women2. Although I don't know of any studies proving that confidence/dominance signals help a businessperson get promoted or a politician get elected, common sense suggests they do. For example, height does have a proven effect in this area, suggesting that our ancestral algorithms for assessing dominance play a major role.

MBlume has already discussed how one cannot simply choose to consciously project dominance signals. The expressions and postures involved are too complicated and too far from the normal domain of conscious control. He suggests using imagination and self-deception to trick the subconscious mind into adopting the necessary role.

So I hope it is not too controversial when I say that subconscious beliefs can significantly affect disease, willpower, physical strength, intelligence, and romantic and financial success.

The second part of my case is that repeating something makes the brain believe it on a subconscious level.

Say Anna Salamon and Steve Rayhawk: "Any random thing you say or do in the absence of obvious outside pressure, can hijack your self-concept for the medium- to long-term future." That's from their excellent post Cached Selves, where they explain that once you say something, even if you don't really mean it, it affects all your beliefs and behaviors afterwards. If you haven't read it yet, read it now: it is one of Less Wrong's growing number of classics.

There's also this study which someone linked me to on Overcoming Bias and to which I keep returning. It demonstrates pretty clearly that we don't have a lot of access to our own beliefs, and tend to make them up based on our behavior. So if I am repeating "I will become a syndicated cartoonist", and my subconscious is not subtle enough to realize I am doing it as part of a complex plot, it might very well assume I am doing it because, well, I think I will become a syndicated cartoonist. And the subconscious quite likes to keep beliefs consistent, so once it "discovers" I have that belief, it may edit whatever it needs to edit to become more consistent with it.

There have been a few studies vaguely related to affirmations. One that came out just a few weeks ago found that minorities who wrote 'value affirmation' essays did significantly better in school (although the same effect did not apply to Caucasians) . Another found that some similar sort of 'value affirmation' decreased stress as measured in cortisol and other physiological measures . But AFAIK no one's ever done a study on Adams-variety simple personal affirmations in all of their counter-intuitive weirdness, probably because it sounds so silly, and I think that's a shame. If they works, it's a useful self-help technique and akrasia-buster. If they don't work, that blocks off a few theories about how the mind works and helps us start looking for alternatives.

 

Footnote

1: A story I like: in one of the studies that discovered the placebo effect for alcohol, one of the male participants who falsely believed he was drunk tried to cop a feel of a female researcher's breasts. That must have been the most awkward debriefing ever.

2: Here I'm not just making my usual mistake and being accidentally sexist; I really mean "pick up women". There is less research suggesting the same thing works on men. See Chapter 6 of The Adapted Mind, "The Evolution of Sexual Attraction: Evaluative Mechanisms in Women".

In response to comment by Yvain on The mind-killer
Comment author: steven0461 03 May 2009 12:11:23AM 5 points [-]

Politics discussion by rationalists is likely to have the most impact when it's about issues that are important, but that aren't widely recognized as such and therefore have relatively few people pulling on the rope. I don't see any point in discussing the Iraq war, say.

In response to comment by steven0461 on The mind-killer
Comment author: Yvain 03 May 2009 12:22:58AM 3 points [-]

Politics action by rationalists is likely to have the most impact on such topics. But since there are already some such topics we know about (global existential risk, for example, or teaching rationality in schools). What do we gain by discovering several more of these and then discussing them?

In response to The mind-killer
Comment author: Yvain 03 May 2009 12:06:41AM 8 points [-]

I agree with ciphergoth that we would probably have an easier time discussing political issues than some other communities, and I agree with HalFinney that it's probably not a very good use of our time anyway. Let's say that everyone on LessWrong agrees on a solution to some political problem. So what? We already have lots of good ideas no one will listen to. It doesn't take a long-time reader of Overcoming Bias to realize marijuana criminalization isn't working so well, but so far the efforts of groups with far more resources than ourselves have been mostly in vain.

If someone came up with a new idea for pulling ropes sideways, that might be useful. For example, Robin's idea of futarchy is interesting, although so large-scale that it would be very hard to implement. If someone came up with a suggestion that brilliant, but for a smaller problem, it might do some good. But I always interpreted our posting policies to permit that sort of thing anyway.

But I think the best thing we could possibly do would be to raise the sanity waterline - a rising tide lifts all boats. That means coming up with compact, attractive summaries of our key findings and spreading them as far as possible. More on this later, possibly.

Comment author: michaelkeenan 29 April 2009 04:25:30AM 0 points [-]

Hey Yvain. I found your blog a little while ago (I think it was from an interesting comment on Patri's LiveJournal, or maybe he linked to you). I disagree that your blog isn't interesting to people that aren't immediate friends (for example, I found your arguments about boycotts and children's rights to be interesting and persuasive). I respect that you seem to not want to link to it here, so I won't. But I urge you to change your mind!

Comment author: Yvain 02 May 2009 11:49:31PM 0 points [-]

Thank you, Michael, for not linking to it here, and thank you, Badger, for the kind words. Although I'm not going to accept any comparisons to EY until I've come up with and implemented at least one feasible plan to save the world.

What I Tell You Three Times Is True

46 Post author: Yvain 02 May 2009 11:47PM

"The human brain evidently operates on some variation of the famous principle enunciated in 'The Hunting of the Snark': 'What I tell you three times is true.'"

   -- Norbert Weiner, from Cybernetics

Ask for a high-profile rationalist, and you'll hear about Richard Dawkins or James Randi or maybe Peter Thiel. Not a lot of people would immediately name Scott Adams, creator of Dilbert. But as readers of his blog know, he's got a deep interest in rationality, and sometimes it shows up in his comics: for example, this one from last week. How many people can expose several million people to the phrase "Boltzmann brain hypothesis" and have them enjoy it?

So I was very surprised to find Adams was a believer in and evangelist of something that sounded a lot like pseudoscience. "Affirmations" are positive statements made with the belief that saying the statement loud enough and long enough will help it come true. For example, you might say "I will become a syndicated cartoonist" fifteen times before bed every night, thinking that this will in fact make you a syndicated cartoonist. Adams partially credits his success as a cartoonist to doing exactly this.

He admits "it sounds as if I believe in some sort of voodoo or magic", and acknowledges that "skeptics have suggested, and reasonably so, that this is a classic case of selective memory" but still swears that it works. He also has "received thousands of e-mails from people recounting their own experiences with affirmations. Most people seem to be amazed at how well they worked."

None of this should be taken too seriously without a controlled scientific study investigating it, of course. But is it worth the effort of a study, or should it be filed under "so stupid that it's not worth anyone's time to investigate further"?

I think there's a good case to be made from within a rationalist/scientific worldview that affirmations may in fact be effective for certain goals. Not miraculously effective, but not totally useless either.

To build this case, I want to provide evidence for two propositions. First, that whether we subconsciously believe we can succeed affects whether or not we succeed. Second, that repeating a statement verbally can make the subconscious believe it.

The link between belief in success and success has progressed beyond the motivational speaker stage and into the scientific evidence stage. The best-known of these links is the placebo effect. For certain diseases, believing that you will get better does increase your probability of getting better. This works not only subjectively (ie you feel less pain) but objectively (ie ulcers heal more quickly, inflammation decreases).

The placebo effect applies in some stranger cases outside simple curative drugs. A placebo stop-smoking pill does increase your chance of successfully quitting tobacco. Placebo strength pills enable you to run faster and lift more weight. Placebo alcohol makes you more gregarious and less inhibited1. Placebo therapies for phobia can desensitize you to otherwise terrifying stimuli.

There are some great studies about the effect of belief in school settings. Pick a student at random and tell the teacher that she's especially smart, and by the end of the year she will be doing exceptionally well; tell the teacher that she is exceptionally stupid, and by the end of the year she'll be doing exceptionally poorly. The experimenters theorized that the teacher's belief about the student's intelligence was subconsciously detected by the student, and that the student was somehow adjusted her performance to fit that belief. In a similar study, minority students were found to do worse on tests when reminded of stereotypes that minorities are stupid, and better when tested in contexts that downplayed their minority status, suggesting that the students' belief that they would fail was enough to make them fail.

Belief can also translate to success when mediated by signals of dominance and confidence. We've already discussed how hard-to-fake signals of confidence can help someone pick up women2. Although I don't know of any studies proving that confidence/dominance signals help a businessperson get promoted or a politician get elected, common sense suggests they do. For example, height does have a proven effect in this area, suggesting that our ancestral algorithms for assessing dominance play a major role.

MBlume has already discussed how one cannot simply choose to consciously project dominance signals. The expressions and postures involved are too complicated and too far from the normal domain of conscious control. He suggests using imagination and self-deception to trick the subconscious mind into adopting the necessary role.

So I hope it is not too controversial when I say that subconscious beliefs can significantly affect disease, willpower, physical strength, intelligence, and romantic and financial success.

The second part of my case is that repeating something makes the brain believe it on a subconscious level.

Say Anna Salamon and Steve Rayhawk: "Any random thing you say or do in the absence of obvious outside pressure, can hijack your self-concept for the medium- to long-term future." That's from their excellent post Cached Selves, where they explain that once you say something, even if you don't really mean it, it affects all your beliefs and behaviors afterwards. If you haven't read it yet, read it now: it is one of Less Wrong's growing number of classics.

There's also this study which someone linked me to on Overcoming Bias and to which I keep returning. It demonstrates pretty clearly that we don't have a lot of access to our own beliefs, and tend to make them up based on our behavior. So if I am repeating "I will become a syndicated cartoonist", and my subconscious is not subtle enough to realize I am doing it as part of a complex plot, it might very well assume I am doing it because, well, I think I will become a syndicated cartoonist. And the subconscious quite likes to keep beliefs consistent, so once it "discovers" I have that belief, it may edit whatever it needs to edit to become more consistent with it.

There have been a few studies vaguely related to affirmations. One that came out just a few weeks ago found that minorities who wrote 'value affirmation' essays did significantly better in school (although the same effect did not apply to Caucasians) . Another found that some similar sort of 'value affirmation' decreased stress as measured in cortisol and other physiological measures . But AFAIK no one's ever done a study on Adams-variety simple personal affirmations in all of their counter-intuitive weirdness, probably because it sounds so silly, and I think that's a shame. If they works, it's a useful self-help technique and akrasia-buster. If they don't work, that blocks off a few theories about how the mind works and helps us start looking for alternatives.

 

Footnote

1: A story I like: in one of the studies that discovered the placebo effect for alcohol, one of the male participants who falsely believed he was drunk tried to cop a feel of a female researcher's breasts. That must have been the most awkward debriefing ever.

2: Here I'm not just making my usual mistake and being accidentally sexist; I really mean "pick up women". There is less research suggesting the same thing works on men. See Chapter 6 of The Adapted Mind, "The Evolution of Sexual Attraction: Evaluative Mechanisms in Women".

Comment author: PhilGoetz 17 April 2009 05:38:35PM *  21 points [-]

This very good post! Yay Yvain! You have high karma. Please give me stock advice.

I know a guy who constructed a 10-dimensional metric space for English words, then did PCA on it. There were only 4 significant components: good-bad, calm-exciting, open-closed, basic-elaborate. They accounted for 65%, 20%, 9%, and 5% of the variance in the 10-dimensional space, leaving 1% for everything else. This means that we need only 8 adjectives in English 99% of the time.

So it would not be surprising to find that our morality is a quick hack on the same machinery that runs our decisions about which food to eat or which pet to adopt.

This could be explored more deeply in another post.

Comment author: Yvain 02 May 2009 10:28:42PM 8 points [-]

Sorry, I didn't see this until today.

Can you give me a link to some more formal description of this? I don't understand how you would use a ten dimensional metric space to capture English words without reducing them to a few broad variables, which seems to be what he's claiming as a result.

Comment author: JulianMorrison 30 April 2009 12:49:47AM 12 points [-]
  • You are so far below having civil rights, that people use you as an excuse to put restrictions on adults.
Comment author: Yvain 01 May 2009 12:28:48PM 0 points [-]

Agreed that children don't have a lot of good reasons to be happy, but not sure that's very relevant here. My observations of children suggest that when they're raised well they have a really high happiness set point, or are just plain better at being happy than adults for developmental reasons unrelated to their circumstances. I don't really trust my own memories of childhood, but the children I hang around always look like they're having more fun than adults nearby, unless they're specifically being prevented from doing so.

Comment author: kurige 29 April 2009 05:28:47PM 2 points [-]

Thanks for the examples of how to apply OB/LW techniques to everyday life.

Definitely more articles in this vein would be greatly appreciated.

Comment author: Yvain 29 April 2009 10:12:40PM 5 points [-]

Agreed. I especially like technique 1. A related technique is to imagine you're going to be called on it. For example, if I predict there's a 90% chance the economic crisis will be over by 2011, I imagine that the second I say the estimate, Omega will come down from the sky and say whether it is or it isn't. Quite often I find that I'm worrying a bit more than 10% that Omega will announce that I was wrong.

Comment author: Kaj_Sotala 29 April 2009 07:19:13AM *  3 points [-]

Great post. This might be the one thing that I'd wish more people would realize.

(Out of curiosity, what were the creative versus ordinary teaching methods you tried? Just wanting to see if I'm a similiar outlier as you.)

Comment author: Yvain 29 April 2009 10:05:19PM 8 points [-]

Keeping in mind that I taught English as a second language to older elementary school children:

Ordinary teaching methods: constant repetition of unconnected topics followed by endless vapid games. For example, a game of bingo with vocabulary words in each square. Attempts to trick children into thinking something was interesting; for example, calling vocabulary "word baseball" or something like that and dressing up in a baseball cap while teaching it.

Things I predicted would work better: attempts to make material genuinely interesting, have each lesson build on the previous, and create links between different concepts. For example, a lesson on the days of the week including a mini-presentation on the Norse gods after whom they were named, references to previous lessons when we had learned "sun" and "moon" for Sunday and Monday. Attempt to teach how to apply general principles instead of doing everything ad hoc.

Comment author: Yvain 29 April 2009 03:50:48PM 4 points [-]

This post made me think of this website.

Comment author: swestrup 29 April 2009 02:50:22AM 11 points [-]

This post completely takes the wind out of the sails of a post I was planning to make on 'Self-Induced Biases' where one mistakes the environment one has chosen for themselves as being, in some sense, 'typical' and then derives lots of bad mental statistics from this. Thus, chess fanatics will tend to think that chess is much more popular than it is, since all their friends like chess, disregarding the fact that they chose those friends (at least partly) based on a commonality of interests.

A worse case is when the police start to think that everyone is a criminal because that's all they ever seem to meet.

Comment author: Yvain 29 April 2009 08:56:10AM 4 points [-]

No, not really. I kind of thought we needed more on that, but that this post was long enough already. And I didn't even think of the police-criminal thing. If you have more than what you said in this comment, please do post it, maybe with this post in the "related to" section.

Comment author: Alicorn 28 April 2009 10:43:51PM *  16 points [-]

This reminds me of some of the literature on fallibility of introspection. (If you have time only for one essay, read "The Unreliability of Naïve Introspection" and try the experiment with the playing card.)

As far as generalizing about an entire gender: It's extremely likely that I know a wildly unrepresentative sample of women, but why would you assume that the pickup artists don't? I imagine they meet vast numbers of women, but if they find them all at parties and clubs and bars, they're going to meet the kinds of women who go to parties and clubs and bars, not the ones who spend their time gardening at home or who go to all-women gyms to avoid being hit on or the ones who play D&D with their brothers in the basement. Even if their statements are accurate about that sort of woman (which I am not yet prepared to believe), that doesn't make them applicable to the entire gender, and the stereotype remains wildly inappropriate and offensive. If you're hearing things about men as a group that don't apply to you or any men you know, then chances are you're not hearing from someone who has a really ideal sample. If a female friend of mine complains about her sixth boyfriend in a row being a jerk, I don't conclude that men are jerks, I conclude that she has terrible taste.

Comment author: Yvain 29 April 2009 08:50:46AM 7 points [-]

You're right.

The success of pickup artist techniques only prove that there are enough women who are vulnerable to them to keep pickup artists in business. Same with any stereotypes about males. If my post implied there was strong evidence that such people were in a majority, that was an error. Although I think if these women were too small of a minority, the PUAs would alter their techniques to ones that worked on a more representative sample of women (assuming they're rational; I don't know any, but people in this community seem to have a high opinion of them.)

I think the general point that we're too unwilling to believe there are significant groups of people who think differently from ourselves still stands, though, whether it's closer to 20% or 60%.

Comment author: scientism 29 April 2009 03:15:19AM *  48 points [-]

Maybe I'm just cynical but I think people vastly overestimate their own goodness. Often "goodness" is just a way to dress up powerlessness. Like an overweight man might say he's "stocky" or an overweight woman might say she's "curvy," so an undesirable or shy man or woman might emphasize the upside: "I would never cheat." There's a version of the typical mind fallacy in there: a person might genuinely think they would never cheat but be extrapolating from a position where the opportunity rarely presents itself. We can all talk about how, if we were in a position of political power, we'd never succumb to bribes or cronyism because we don't have any political power. It both makes us look good and, as far as we know, it's true. I think testimony, especially when it comes to ones moral worth, is the least valuable form of data available.

Comment author: Yvain 29 April 2009 08:47:25AM 1 point [-]

But we also have evidence from our past actions. For example, I have never cheated on a test or shoplifted in the past, so I assume this is true of everyone. My friends say the same thing (and I mostly believe them).

Generalizing From One Example

259 Post author: Yvain 28 April 2009 10:00PM

Related to: The Psychological Unity of Humankind, Instrumental vs. Epistemic: A Bardic Perspective

"Everyone generalizes from one example. At least, I do."

   -- Vlad Taltos (Issola, Steven Brust)

My old professor, David Berman, liked to talk about what he called the "typical mind fallacy", which he illustrated through the following example:

There was a debate, in the late 1800s, about whether "imagination" was simply a turn of phrase or a real phenomenon. That is, can people actually create images in their minds which they see vividly, or do they simply say "I saw it in my mind" as a metaphor for considering what it looked like?

Upon hearing this, my response was "How the stars was this actually a real debate? Of course we have mental imagery. Anyone who doesn't think we have mental imagery is either such a fanatical Behaviorist that she doubts the evidence of her own senses, or simply insane." Unfortunately, the professor was able to parade a long list of famous people who denied mental imagery, including some leading scientists of the era. And this was all before Behaviorism even existed.

The debate was resolved by Francis Galton, a fascinating man who among other achievements invented eugenics, the "wisdom of crowds", and standard deviation. Galton gave people some very detailed surveys, and found that some people did have mental imagery and others didn't. The ones who did had simply assumed everyone did, and the ones who didn't had simply assumed everyone didn't, to the point of coming up with absurd justifications for why they were lying or misunderstanding the question. There was a wide spectrum of imaging ability, from about five percent of people with perfect eidetic imagery1 to three percent of people completely unable to form mental images2.

Dr. Berman dubbed this the Typical Mind Fallacy: the human tendency to believe that one's own mental structure can be generalized to apply to everyone else's.

He kind of took this idea and ran with it. He interpreted certain passages in George Berkeley's biography to mean that Berkeley was an eidetic imager, and that this was why the idea of the universe as sense-perception held such interest to him. He also suggested that experience of consciousness and qualia were as variable as imaging, and that philosophers who deny their existence (Ryle? Dennett? Behaviorists?) were simply people whose mind lacked the ability to easily experience qualia. In general, he believed philosophy of mind was littered with examples of philosophers taking their own mental experiences and building theories on them, and other philosophers with different mental experiences critiquing them and wondering why they disagreed.

The formal typical mind fallacy is about serious matters of mental structure. But I've also run into something similar with something more like the psyche than the mind: a tendency to generalize from our personalities and behaviors.

For example, I'm about as introverted a person as you're ever likely to meet - anyone more introverted than I am doesn't communicate with anyone. All through elementary and middle school, I suspected that the other children were out to get me. They kept on grabbing me when I was busy with something and trying to drag me off to do some rough activity with them and their friends. When I protested, they counter-protested and told me I really needed to stop whatever I was doing and come join them. I figured they were bullies who were trying to annoy me, and found ways to hide from them and scare them off.

Eventually I realized that it was a double misunderstanding. They figured I must be like them, and the only thing keeping me from playing their fun games was that I was too shy. I figured they must be like me, and that the only reason they would interrupt a person who was obviously busy reading was that they wanted to annoy him.

Likewise: I can't deal with noise. If someone's being loud, I can't sleep, I can't study, I can't concentrate, I can't do anything except bang my head against the wall and hope they stop. I once had a noisy housemate. Whenever I asked her to keep it down, she told me I was being oversensitive and should just mellow out. I can't claim total victory here, because she was very neat and kept yelling at me for leaving things out of place, and I told her she needed to just mellow out and you couldn't even tell that there was dust on that dresser anyway. It didn't occur to me then that neatness to her might be as necessary and uncompromisable as quiet was to me, and that this was an actual feature of how our minds processed information rather than just some weird quirk on her part.

"Just some weird quirk on her part" and "just being oversensitive" are representative of the problem with the typical psyche fallacy, which is that it's invisible. We tend to neglect the role of differently-built minds in disagreements, and attribute the problems to the other side being deliberately perverse or confused. I happen to know that loud noise seriously pains and debilitates me, but when I say this to other people they think I'm just expressing some weird personal preference for quiet. Think about all those poor non-imagers who thought everyone else was just taking a metaphor about seeing mental images way too far and refusing to give it up.

And the reason I'm posting this here is because it's rationality that helps us deal with these problems.

There's some evidence that the usual method of interacting with people involves something sorta like emulating them within our own brain. We think about how we would react, adjust for the other person's differences, and then assume the other person would react that way. This method of interaction is very tempting, and it always feels like it ought to work.

But when statistics tell you that the method that would work on you doesn't work on anyone else, then continuing to follow that gut feeling is a Typical Psyche Fallacy. You've got to be a good rationalist, reject your gut feeling, and follow the data.

I only really discovered this in my last job as a school teacher. There's a lot of data on teaching methods that students enjoy and learn from. I had some of these methods...inflicted...on me during my school days, and I had no intention of abusing my own students in the same way. And when I tried the sorts of really creative stuff I would have loved as a student...it fell completely flat. What ended up working? Something pretty close to the teaching methods I'd hated as a kid. Oh. Well. Now I know why people use them so much. And here I'd gone through life thinking my teachers were just inexplicably bad at what they did, never figuring out that I was just the odd outlier who couldn't be reached by this sort of stuff.

The other reason I'm posting this here is because I think it relates to some of the discussions of seduction that are going on in MBlume's Bardic thread. There are a lot of not-particularly-complimentary things about women that many men tend to believe. Some guys say that women will never have romantic relationships with their actually-decent-people male friends because they prefer alpha-male jerks who treat them poorly. Other guys say women want to be lied to and tricked. I could go on, but I think most of them are covered in that thread anyway.

The response I hear from most of the women I know is that this is complete balderdash and women aren't like that at all. So what's going on?

Well, I'm afraid I kind of trust the seduction people. They've put a lot of work into their "art" and at least according to their self-report are pretty successful. And unhappy romantically frustrated nice guys everywhere can't be completely wrong.

My theory is that the women in this case are committing a Typical Psyche Fallacy. The women I ask about this are not even remotely close to being a representative sample of all women. They're the kind of women whom a shy and somewhat geeky guy knows and talks about psychology with. Likewise, the type of women who publish strong opinions about this on the Internet aren't close to a representative sample. They're well-educated women who have strong opinions about gender issues and post about them on blogs.

And lest I sound chauvinistic, the same is certainly true of men. I hear a lot of bad things said about men (especially with reference to what they want romantically) that I wouldn't dream of applying to myself, my close friends, or to any man I know. But they're so common and so well-supported that I have excellent reason to believe they're true.

This post has gradually been getting less rigorous and less connected to the formal Typical Mind Fallacy. First I changed it to a Typical Psyche Fallacy so I could talk about things that were more psychological and social than mental. And now it's expanding to cover the related fallacy of believing your own social circle is at least a little representative of society at large, which it very rarely is3.

It was originally titled "The Typical Mind Fallacy", but I'm taking a hint fromt the quote and changing it to "Generalizing From One Example", because that seems to be the link between all of these errors. We only have direct first-person knowledge one one mind, one psyche, and one social circle, and we find it tempting to treat it as typical even in the face of contrary evidence.

This, I think, is especially important for the sort of people who enjoy Less Wrong, who as far as I can tell are with few exceptions the sort of people who are extreme outliers on every psychometric test ever invented.


Footnotes

1. Eidetic imagery, vaguely related to the idea of a "photographic memory", is the ability to visualize something and have it be exactly as clear, vivid and obvious as actually seeing it. My professor's example (which Michael Howard somehow remembers even though I only mentioned it once a few years ago) is that although many people can imagine a picture of a tiger, only an eidetic imager would be able to count the number of stripes.

2. According to Galton, people incapable of forming images were overrepresented in math and science. I've since heard that this idea has been challenged, but I can't access the study.

3. The example that really drove this home to me: what percent of high school students do you think cheat on tests? What percent have shoplifted? Someone did a survey on this recently and found that the answer was nobhg gjb guveqf unir purngrq naq nobhg bar guveq unir fubcyvsgrq (rot13ed so you have to actually take a guess first). This shocked me and everyone I knew, because we didn't cheat or steal during high school and we didn't know anyone who did. I spent an afternoon trying to find some proof that the study was wrong or unrepresentative and coming up with nothing.

In response to Bayesian Cabaret
Comment author: Robin 28 April 2009 03:59:21AM 3 points [-]

I'm not sure if this is an event to celebrate, I think less posts by Eliezer is a bad thing.

In response to comment by Robin on Bayesian Cabaret
Comment author: Yvain 28 April 2009 11:21:53AM 8 points [-]

It's...bittersweet. Like a graduation ceremony. It's nice to know you've learned a lot and reached a level where you can go out on your own. It's sad that you've reached the end of what formal instruction will teach you and you're going to have to part ways with your teachers and the college community. And it's scary that you've got to go out into the real world when you still know so little.

Comment author: Yvain 28 April 2009 11:18:54AM 3 points [-]

Another "thank you". It's been...remarkable.

I hope you can continue to use your symbolic position to keep us cohesive for a while even when you're not posting at quite the same rate. And I look forward to seeing more of your often very interesting ad hoc observational posts now that you don't feel compelled to stick to a sequence.

Bayesian Cabaret

19 Yvain 27 April 2009 10:29PM

I'd heard rumors that some leading Bayesians had achieved rank in the Bardic Conspiracy. But I wasn't aware that every two years, some of the world's top statisticians hold a Bayesian Cabaret, full of songs, dances, and skits about Bayesian probability theory.

...no, really. Really. I think my favorite has got to be this one.

YouTube seems to be full of this stuff, including What A Bayesian World and We Didn't Start The Prior. Be sure to also check out some of the recordings and the Bayesian Songbook.

Eliezer's finished his sequences, and it's a new era for Less Wrong. Let's celebrate...Bayesian style!

Comment author: Yvain 27 April 2009 06:36:57PM *  69 points [-]

This post raises a whole constellation of connected questions, so here are my thoughts on all of them:

If the question is "Can Wednesday be religious and still be a smart person who's good at using rationality?", the answer is empirically yes (eg Robert Aumann).

If the question is "Can we still call Wednesday rational if she's religious?" the answer is to taboo "rational" and let the problem take care of itself.

If the question is "Is it okay for Wednesday to be religious?" the question is confused in the first place and any answer would be equally confused.

If the question is "Should Wednesday choose to believe religion?" the answer is that you don't voluntarily choose your beliefs so it doesn't matter.

If the question is "Should Wednesday, while not exactly choosing to believe religion, avoid thinking about it too hard because she thinks doing so will make her an atheist?," then she's already an atheist on some level because she thinks knowing more will make her more atheist, which implies atheism is true. This reduces to the case of deception, which you seem to be against unconditionally.

If the question is "Should I, as an outside observer, do my best to convince Wednesday religion is wrong?" the answer depends on your moral system. I'm a utilitarian, so I would say no - I think it's a background assumption here that she's happier being deceived. I know you're not a utilitarian, so you'd have to work it out in whatever system you use.

If the question is "Should we at Less Wrong exclude all theists?", my answer is of course not. If they want to come here and talk about prisoners dilemmas or the Singularity or something, then of course we should welcome their opinions.

If the question is "Should we at Less Wrong tell all theists they can't talk about how great religion is?" my answer is a qualified "yes". Not because we specifically hate religion, but for the same reason we don't allow posts explicitly about politics. There are places for those debates, this isn't one of those places, and having them completely changes the feel of a community and saps its energy.

If the question is "Should we at Less Wrong stop acting like atheism is an open-and-shut case?," my answer is "no". Sometimes in order to move on, we've got to accept certain assumptions. For example, even though there are a few hard-core steady state theorists out there, most astronomers have accepted the Big Bang as a default assumption because they can get more done by building on Big Bang theory and working out its exact implications then they can debating the last few steady-staters ad nauseum or refusing to even mention the beginning of the universe because it might exclude someone. Christians work in exactly the same way; when they want to discuss obscure points of theology, they start from the assumption that God exists and work from there, although they'll discard that assumption when they're debating an atheist. I don't hold it against these Christians - they'd hardly be able to do theology without it - and I hope they don't hold it against us.

If the question is "Should we at Less Wrong stop saying mean things about religion?" then my answer is that we should never deliberately say mean things just for the sake of saying mean things, but that if it's absolutely necessary to condemn religion to make some greater point (like to use it as an example of a bias towards anthropomorphism) then it's not worth refraining from it to prevent potentially some hypothetical theist from feeling excluded. However, writers should make sure to phrase it as neutrally and non-insultingly as possible, something atheists are generally bad at.

If the question is "What kind of person would name their daughter Wednesday?", I have no good answer. Maybe someone who really, really liked the Thursday Next books?

Also, this wins my prize for most intriguing title on LW so far.

Comment author: Cameron_Taylor 27 April 2009 03:45:30PM 1 point [-]

When considering the impact on your success and quality of life, how useful is a dedicated emphasis on improving rationality to you? - Dramatically improved my life - Somewhat useful - Irrelevant - Somewhat of a hindrance - A significant dissadvantage

Comment author: Yvain 27 April 2009 03:56:38PM 0 points [-]

I thought Anna already covered that very well. Is there some reason you want to know this as part of an interaction with the other questions on the survey?

Comment author: MichaelBishop 27 April 2009 05:02:44AM 3 points [-]

nutritional supplements, diet, exercise habits, height and weight

Comment author: Yvain 27 April 2009 03:03:35PM 0 points [-]

Please tell me the exact questions you want. Also, why height and weight? I appreciate wanting to know what effect diet has on weight, but that's way beyond the scope of this survey.

Comment author: gjm 27 April 2009 09:58:46AM 0 points [-]

1. A question posed simply in terms of "the best level" would be measuring some sort of tangled-up combination of respondents' values and their opinions about facts. That might be a bad thing (though I note that the question about political affiliation, at least, has the same feature). Instead, one could ask something like "what level of aid do you think would maximize Africa's GDP after 20 years?" or "what level of aid do you think would maximize average expected QALYs at birth over the whole human population".

2. When considering an individual's charitable activity, of course we should think in terms of marginal efficiencies. That's not so clear when considering the question of the total amount of aid that might go from the affluent West to the Third World.

3. You mean (unless you have relevant information I don't, which is eminently possible) that some African economists are saying that the aid is harmful. It would be much more interesting to know typical African economists' opinions. If nothing else, there is obvious sampling bias here: if two African economists approach an American publisher, one proposing to write a book saying "Aid is actively harmful; stop it now" and one proposing to write one saying "Aid is useful; please do a bit more of it", which one is going to get the contract? It seems to me that there are multiple different factors making it far more likely to be the first one that have scarcely any correlation with the actual truth of the matter.

4. Yes, of course, actual decisions need to be made project by project. That doesn't mean that one can't hold an opinion about the approximate gross amount of aid there should be. (Such as, for instance, "none", which is an opinion you don't seem to object to even though it's the ultimate in not-project-by-project answers since it necessarily returns the same answer for every project.)

Comment author: Yvain 27 April 2009 12:12:59PM 3 points [-]

How would everyone feel about a question phrased something like:

"True or false: the marginal effect of extra money being given to aid in Africa through a charity like UNICEF is generally positive."

Comment author: gjm 26 April 2009 11:42:59PM 2 points [-]

Robin Hanson notoriously thinks that most medicine does little or no good. I'd guess that he opposes large-scale socialized medicine on these grounds, though that's not a foregone conclusion and I don't think I've seen an explicit statement from him about this. It's probably more usual to think that medicine is great and we should all have easier access to more of it. How about a question somewhere in this vicinity?

Comment author: Yvain 27 April 2009 06:25:46AM 1 point [-]

Yes, but how can we phrase this rigorously? "Medicine does little good" seems too open to interpretation.

Comment author: Vladimir_Nesov 26 April 2009 10:10:45PM *  4 points [-]
  • "Time per day on OB/LW" is hard to measure, since I'm just being online, studying and working in parallel.
  • "Political views" -- I'd like "not commited" as an option.
  • "Santa" -- understanding your position is a process, so e.g. clear-cut "yes/no" doesn't map on my "I contemplated the notion, and was unsure before growing old/perceptive enough to realize it's a running joke".
  • Before the questions on probabilities, it'd be nice to ask about the position on interpretation of probability.
  • There should be questions about position on morality. I suggest: consequentialist (in Yudkowsky's interpretation/other), hedonist, others?
  • Another question: do you hold an explicit utilitarian position (on a form of preference order) such as average or total utilitarianism.
  • The question on whether the data given by the person who fills in the survey should be publicly available should be included in the survey.
  • I'd like to separate Sci-fi from fantasy.
  • Add questions about habits of learning: do you learn technical stuff unrelated to work.
  • What do people do in leisure time (watch TV/serf Internet/solve crosswords/study math).
  • Technically, "Cooperate" in a standard PD is an incorrect answer, since the fact that you know that the other one is a Cooperator is not built into the problem.
  • Don't call that hideous scheme you set up "probability". The log score will punish you infinitely for this heresy.
  • A question about procrastination
  • A question about diet
  • Knowledge of related math: logic, probability theory, Bayesian networks, inference algorithms, expected utility, microeconomics, causal/evidential decision theories.
  • Knowledge of biases/ev-psych literature: read stuff on OB/LW, read a serious book, read (how many) papers.
Comment author: Yvain 26 April 2009 10:27:24PM 5 points [-]

I specifically excluded "not committed" as an option on the political views section, because a lot of rationalists have a tendency to go towards "not committed" to signal how they're not blind followers of a party, when really they have very well defined political views.

I, for example, absolutely refuse to register with a political party, answer "independent" to any questions about my political affiliation, talk a good talk about how both parties are equally crooks, and then proceed to vote for the Democrat nine times out of ten. I would kind of like to force people like me to put "Democrat" on there so that we get more data.

I will change this if enough people agree with Vladimir.

Comment author: TheThinKing 26 April 2009 10:16:49PM 0 points [-]

There is plenty of literature out there about how groups can go wrong. We need to make sure we do not fall victim to those traps. What are ways we can identify and avoid known pitfalls?

Perhaps we should include some questions about the perception of the community: diversity of viewpoints, strength of conformity, how much you personally identify with the group, things of that nature. These answers could be useful for self-diagnostic purposes, both for the group itself as well as its individual members comparing their answers against others in the group.

Comment author: Yvain 26 April 2009 10:24:29PM 0 points [-]

I'm not sure what you mean. Can you suggest some?

Excuse me, would you like to take a survey?

12 Yvain 26 April 2009 09:23PM

Related to: Practical Rationality Questionnaire

Here among this community of prior-using, Aumann-believing rationalists, it is a bit strange that we don't have any good measure of what the community thinks about certain things.

I no longer place much credence in raw majoritarianism: the majority is too uneducated, too susceptible to the Dark Arts, and too vulnerable to cognitive biases. If I had to choose the people whose mean opinion I trusted most, it would be - all of you.

So, at the risk of people getting surveyed-out, I'd like to run a survey on the stuff Anna Salamon didn't. Part on demographics, part on opinions, and part on the interactions between the two.

I've already put up an incomplete rough draft of the survey I'd like to use, but I'll post it here again. Remember, this is an incomplete rough draft survey. DO NOT FILL IT OUT YET. YOUR SURVEY WILL NOT BE COUNTED.

Incomplete rough draft of survey

Right now what I want from people is more interesting questions that you want asked. Any question that you want to know the Less Wrong consensus on. Please post each question as a separate comment, and upvote any question that you're also interested in. I'll include as many of the top-scoring questions as I think people can be bothered to answer.

No need to include questions already on the survey, although if you really hate them you can suggest their un-inclusion or re-phrasing.

Also important: how concerned are you about privacy? I was thinking about releasing the raw data later in case other people wanted to perform their own analyses, but it might be possible to identify specific people if you knew enough about them. Are there any people who would be comfortable giving such data if only one person were to see the data, but uncomfortable with it if the data were publically accessible?

Comment author: PhilGoetz 26 April 2009 04:37:06PM 0 points [-]

You may find some statistical effect, but why not give them all the same test, and rank them on their actual knowledge of biology and medicine?

Comment author: Yvain 26 April 2009 05:48:14PM 0 points [-]

Seems too potentially confounding. How would you distinguish between:

a) A person who knows a lot about biology wants to promote scientific values in order to become a more valuable person.

and b) A person who knows a lot about biology is smart enough not to believe pseudoscience involving biology?

Comment author: Alicorn 25 April 2009 08:41:21PM 6 points [-]

It is possible that my distaste for the subject has led me to be insufficiently familiar with the intricacies of the motivations of pickup artists. What I have seen (I read this blog, and otherwise have only passing knowledge) leaves me with a very bad taste in my mouth about the practitioners, their attitude towards my gender, and the revolting dishonesty of the entire genre of interaction.

That having been said, it's possible I'm an outlier. Maybe the fact that I don't generally hang out in bars or attend parties has left me with too high an opinion of the sort of women who can be found in those places. Maybe they're just as bad. I don't know. This is exactly the kind of uninformed, emotional shuddering I suspected no one would be interested in.

Comment author: Yvain 26 April 2009 04:01:05PM *  36 points [-]

When I lived in Asia. I would bow to people, be extremely deferential to my superiors, and avoid saying any original thoughts out loud in any situation where I was not the highest status person. I didn't do this because that's Really Deep Down Who I Am, I did it because I read a book on dealing with Asian people, and that was what you were supposed to do. As a result, I got along with the Asians I knew and had pretty good relationships with most of them. If I'd been completely direct and honest all the time, the Asians wouldn't have "appreciated my honesty". They'd have fired me from my job and stayed away from me.

I don't feel guilty for "manipulating" any Asians. I did what I had to do to be successful in Asia, it made me happy, and it made the Asians who worked with me happy.

I interact every day with two groups of people whose ways I find even stranger than the Asians', those being extroverts and women. I basically coexist with extroverts the same way I coexisted with Asians; I read books about their behavior, I figure out what I need to do to get along with them, and I do it. Do I wish I could win their friendship solely by being myself? Yeah. But that was what I tried for about fifteen years, it ended up with me being unhappy and friendless, and instead of me blaming the extroverts for it I decided to learn techniques to get along with them. I think it makes us all better off.

I have split feelings about the seduction stuff. As a "how to trick stupid girls into sleeping with you so you can dump them later ha ha" sort of thing, it is clearly evil. But when I think of it as a guide to dealing with romance in the same way I've already used guides to dealing with Asians and extroverts, well, I could kind of use something like that.

I guess the difference is that the only thing I consider morally wrong is making other people unhappy. To trick a woman who really doesn't like me into having a one-night stand she'll regret later - that's bad. But if there's a woman whom I think I could have a really good relationship with that would make us both very happy, and the only thing stopping her from going out with me is that my body language is unattractive and I don't know how to ask right, then I wouldn't feel too bad about counteracting the stupid tricks her brain is using to prevent her from going out with me with stupid tricks to make her want to.

(disclaimer: this is all probabilistic. There are a few Asians, extroverts, and women whom I have a much easier time getting along with, but in general I find these categories of people harder to understand.)

Comment author: PhilGoetz 26 April 2009 01:34:36PM 12 points [-]

The martian pyramid theory doesn't hold any attraction for me. Maybe it's because I'm supremely rational. But maybe it's because, if someone really discovered a pyramid built on Mars, it would be extremely... irritating.

I've got decades of hard work invested in learning this whole big scheme of how the world works. The Martian pyramid would knock a lot of it down. And then I'd hardly be any better than anybody else.

Comment author: Yvain 26 April 2009 03:14:01PM *  9 points [-]

That's an very good point, especially because most smart people who really understand science share your opinions

I like your last point - that you feel better than everyone else now, because you do get real science, and so you'd hate to have everything change around. Maybe the people who do like.the idea of a Martian pyramid are those who currently feel worse than everyone else, but who would suddenly become better than everyone else if the pyramid were proven real, because they're the only ones who have studied it.

That would also explain why I find some pseudohistory interesting but most pseudoscience just plain annoying.

Possible experiment to test this: give two randomly assigned groups a test on biology and medicine. Give one group a really easy test and tell them they're within the top 10% regarding medical knowledge; give other people a really hard test and tell them they're within the bottom 10% of medical knowledge. Then give both of them an article on some ancient natural alternative medicine treatment and see which group is more positive towards it. I predict the people convinced they know nothing about medicine will support the alternative treatment more, regardless of how much they actually know.

Comment author: Alicorn 26 April 2009 01:35:42AM *  2 points [-]

1) Women with small breasts have no problem breastfeeding, although they do have to do it more frequently - it's possible that in some environments infrequent nursing was an advantage, although I'm skeptical that this alone could be responsible for the male fixation on breasts over other indicators.

2) A footnote or parenthetical saying something like "Note: The word "you" actually refers to you only if you are a straight or bisexual male" would be fine.

I have no special knowledge on how same-sex partners are generally selected, just my data point from being bi myself.

Comment author: Yvain 26 April 2009 01:40:03AM 1 point [-]

Gah. I understand now. Second-person writing strikes again. Edited to third person, and all references to breasts removed and replaced with references to wide hips, which as far as I know everyone agrees have a clear evolutionary benefit.

Comment author: outlawpoet 26 April 2009 01:20:28AM 1 point [-]

As a total sidenote, your choice of examples is bad. If someone solved photosynthesis in a way that output useful engineerable technologies, it would change your life, and the lives of almost everybody else.

Solar power cheap and powerful enough to run most of our technology would be a massive sea change.

Comment author: Yvain 26 April 2009 01:33:21AM *  7 points [-]

No, I don't think the choice of examples is bad - I had another draft where I used understanding the pathogenesis of some common disease as an example, which is even more clearly beneficial.

My point is that even when rational analysis tells us that something will be very useful, the "sense of curiosity" can disagree. Otherwise, we'd all be fascinated by immunology because of its high probability of giving us a cure for cancer and AIDS. Likewise, discovering that Stonehenge was built by aliens would be practically useless unless it provided some way of contacting the aliens or using their technology, but it would still be considered "interesting".

That's why I didn't include "gives a practical benefit" as a criterion. Instead I said "changes a lot of beliefs", which a better understanding of photosynthesis wouldn't, and "teaches you something that other people want to know", which photosynthesis again wouldn't (lots of people would want the improved solar technology, but not many people would care how it worked).

Comment author: Alicorn 26 April 2009 01:07:59AM 4 points [-]

A couple things, both somewhat peripheral to the main point:

1) Is there actually some adaptive benefit to large breasts? I seem to remember reading somewhere that there isn't and they're only common in humans because guys select for them.

2) This is an awfully straight-male-centric post. I realize that this is a community which probably consists mostly of straight (or at least bi) males, but I would have appreciated some explicit parenthetical remark or footnote noting that assumption.

Comment author: Yvain 26 April 2009 01:27:46AM *  5 points [-]

1) I've heard some people say that it involves ability to produce milk for the baby, and others say that it's a signal of health (ie evolution wouldn't concentrate resources there unless all the other organs were already healthy). I only mentioned the first possibility in this post, which on second thought was overly simplistic.

2) You mean in relation to the evo psych of lust? Yes, I suppose that's true; as I said, a lot of this post is based on introspection, and when I introspect about the sex drive I tend to think about the straight male sex drive for obvious reasons. I don't really know what kind of footnote I could include besides "And females also exist and also fall in love for stupid reasons sometimes", but if there's something specific which you think needs to be said I'll add it in.

(AFAIK, there's no good research or theorizing on the evolutionary psychology of how homosexuals select partners, and the evolution of homosexuality is still a confusing and controversial field. Would be interested if you know of anything.)

Comment author: Yvain 26 April 2009 01:06:38AM *  1 point [-]

Maybe you know of the "sincere smile" and the "insincere smile". The sincere smile uses a different set of muscles, most famously the orbicularis oculi, which is somewhere between very hard and impossible to contract voluntarily. As far as I know, the only way to get a genuine smile using that muscle is to either be happy, or pretend to be happy so successfully that you become happy.

When you say that instead of pretending to be dominant you should replicate the behaviors of a dominant person, well, that makes sense, but I think you'd need to prove beforehand that these can be voluntarily replicated in a believable way, and that it wouldn't take more effort and self-control than it's worth.

People are pretty good at picking up microexpressions and other things on an unconscious level; a lot of people can tell the difference between a sincere and insincere smile without being able to verbalize what they're looking for. I wouldn't be surprised if women could tell the difference between someone who's confident and someone who's faking confidence.

Where's Your Sense of Mystery?

35 Post author: Yvain 26 April 2009 12:45AM

Related to: Joy in the Merely Real, How An Algorithm Feels From Inside, "Science" As Curiosity-Stopper

Your friend tells you that a certain rock formation on Mars looks a lot like a pyramid, and that maybe it was built by aliens in the distant past. You scoff, and respond that a lot of geological processes can produce regular-looking rocks, and in all the other cases like this closer investigation has revealed the rocks to be completely natural. You think this whole conversation is silly and don't want to waste your time on such nonsense. Your friend scoffs and asks:

"Where's your sense of mystery?"


You respond, as you have been taught to do, that your sense of mystery is exactly where it should be, among all of the real non-flimflam mysteries of science. How exactly does photosynthesis happen, what is the relationship between gravity and quantum theory, what is the source of the perturbations in Neptune's orbit? These are the real mysteries, not some bunkum about aliens. And if we cannot learn to take joy in the merely real, our life will be empty indeed.

But do you really believe it?

I loved the Joy in the Merely Real sequence. But it spoke to me because it's one of the things I have the most trouble with. I am the kind of person who would have much more fun reading about the Martian pyramid than about photosynthesis.

And the one shortcoming of Joy in the Merely Real was that it was entirely normative, and not descriptive. It tells me I should reserve my sense of mystery for real science, but doesn't explain why it's so hard to do so, or why most people never even try.

So what is this sense of mystery thing anyway?

I think the sense of mystery (sense of wonder, curiosity, call it what you want) is how the mind's algorithm for determining what problems to work on feels from the inside. Compare this to lust, how the mind's algorithm for determining what potential mates to pursue feels from the inside. In both cases, the mind makes a decision based on criteria of its own, which is then presented to the consciousness in the form of an emotion. And in both cases, the mind's decision is very often contrary to our best interest - as anyone who's ever fallen for a woman based entirely on her looks can tell you.

What sort of stuff makes us curious? I don't have anything better than introspection to go on, but here are some thoughts:

1. We feel more curious about things that could potentially alter many different beliefs.
2. We feel more curious about things that we feel like we can solve.
3. We feel more curious about things that might give us knowledge other people want but don't have.
4. We feel more curious about things that use the native architecture; that is, the sorts of human-level events and personal interactions our minds evolved to deal with.

So let's go back and consider how the original example - a pyramid on Mars versus photosynthesis - fits each of these criteria:

The pyramid on Mars could alter our worldview completely1. We'd have to rework all of our theories about ancient history, astronomy, the origin of civilization, maybe even religion. Learning exactly how photosynthesis works, on the other hand, probably won't make too big a difference. I assume it probably involves some sort of chemistry that sounds a lot like the other chemistry I know. I anticipate that learning more about photosynthesis wouldn't alter any of my beliefs except those directly involving photosynthesis and maybe some obscure biochemical reactions.

Pseudoscience and pseudohistory feel solveable. When you're reading a good pseudoscience book, it feels like you have all the clues and you just have to put them together. If you don't believe me, Google some pseudoscience. You'll find hundreds of webpages by people who think they've discovered the 'secret'. One person who says the pyramid on Mars was made by Atlanteans, another who says it was made by the Babylonian gods, another who says it was made by God to test our faith. On the other hand, I know I can't figure out photosynthesis without already being an expert in chemistry and biology. There's not that tantalizing sense of "I could be the one to figure this out!"

Knowing about a pyramid on Mars means you know more than other people. Most of humankind doesn't think there are any structures on Mars - the fools! And if you were to figure it out, you'd be...one of the greatest scientists ever. The one who proved the existence of intelligent life on other planets. It'd be great! In comparison, knowing about photosynthesis makes you one of a few thousand boring chemist types who also know about photosynthesis. Even if you're the first person to discover something new about it, the only people likely to care are...a few thousand boring chemist types.

And the pyramid deals in human-level problems: civilizations, monuments, collapse. Photosynthesis is a matter of equations and chemical reactions; much harder for most people.

Evolutionarily, all these criteria make sense. Of course you should spend more time on a problem if you're likely to solve it and the solution will be very important. And when you're a hunter-gatherer, all your problems are going to be on the human level, so you might as well direct your sense of mystery there. But the algorithm is unsuited to modern day science, when interesting discoveries are usually several inferential distances away in highly specialized domains and don't directly relate to the human level at all.

Again, compare this to lust. In the evolutionary era, mating with a woman with wide hips was quite adaptive for a male. Nowadays, with the advent of the Caesarian section, not so much. Nowadays it's probably most important for him to choose a mate whom he can tolerate for more than a few years so he doesn't end up divorced. But the mental algorithms whose result outputs as lust don't know that, so they end up making him weak-kneed for some wide-hipped woman with a terrible personality. This isn't something to feel guilty about. It's just something he needs to be wary of and devote some of his willpower resources toward fighting.

The practical take home advice, for me at least, is to treat curiosity in the same way. For a while, I felt genuinely guilty about my attraction to pseudohistory, as if it was some kind of moral flaw. It's not, no more than feeling lust towards someone you don't like is a moral flaw. They're both just misplaced drives, and all you can do is ignore, sublimate, or redirect them2.

The great thing about lust is that satisfying your unconscious and conscious feelings don't have to be mutually exclusive. Sometimes somebody comes around who's both beautiful and the sort of person you want to spend the rest of your life with. Problem solved. Other times, once your conscious mind commits to someone, your unconscious mind eventually starts coming around. These are the only two solutions I've found for the curiosity problem too.

The other practical take home advice here is for anyone whose job is educating others about science. Their job is going to be a lot easier if they can take advantage of this sense of mystery. The best science teachers I know do this. They emphasize the places where science produces counterintuitive, worldview-changing results. They present their information in the form of puzzles just difficult enough for their students to solve with a bit of effort. They try to pique their students interest with tales of the unusual or impressive. And they try to use metaphors to use the native architecture of human minds: talking about search algorithms in terms of water flowing downhill, for example.

I hope that any work that gets done on Less Wrong involving synchronizing conscious and unconscious feelings and fighting akrasia can be applied to this issue too.

 

Footnotes:

1: The brain seems generally bad at dealing with tiny probabilities of huge payoffs. It may be that the payoff measured in size of paradigm shift from any paranormal belief being true is just so high that people aren't very good at discounting for the very small percent chance of it being true.

2: One big question I'm still uncertain about: why do some people, despite it all, find science really interesting? How come this is sometimes true of one science and not others? I have a friend who loves physics and desperately wants to solve its open questions, but whose eyes glaze over every time she hears about biology - what's up with that?

Comment author: Yvain 19 April 2009 05:20:10PM 9 points [-]

I like this game, but you've got to make it interesting. In that vein, here's mine:

1) Eliezer once wrote an Overcoming Bias post where he said something like "Make one little mistake in a whole long chain of inferences, and you end up in Outer Mongolia." The first time I read this post was from a webcafe in Outer Mongolia.

2) I have pet several full-grown tigers, and one of them almost fell asleep on my lap.

3) I once broke up with a girl partly because she was a deontologist.

4) One of my school teachers once accused me, in all seriousness, of being a witch. Not witch as in "Wiccan", but witch as in "has magic powers and consorts with the devil".

5) I was given a blessing by a Tibetan lama in the mountains of Nepal.

Comment author: Yvain 26 April 2009 12:15:10AM 1 point [-]

Gur snyfrubbq vf guerr.

Ahzore bar vf gehr; V jnf va Nfvn ng gur gvzr, nyjnlf jnagrq gb frr Zbatbyvn, naq fcrag gur jrrx va Hyna Ongne naq fheebhaqvatf.

Ahzore gjb vf nyfb gehr: frr uggc://ra.jvxvcrqvn.bet/jvxv/Gvtre_Grzcyr.

Ahzore guerr vf snyfr: V qvq tb ba BXPhcvq, frr fbzrbar jub fnvq va ure cebsvyr gung fur jnf n qrbagbybtvfg, naq guvax "Vg'q arire jbex", ohg gung'f nf sne nf V'ir tbggra.

Ahzore sbhe vf gehr; vg jnf n penml byq ernyyl eryvtvbhf fhofgvghgr grnpure jub fgnegrq bss trarenyyl pbaqrzavat jvgpupensg, V qrpvqrq gb nethr jvgu uvz orpnhfr V nz trarenyyl nethzragngvir, naq uvf penmvarff qvq gur erfg.

Ahzore svir vf gehr: frr uggc://ra.jvxvcrqvn.bet/jvxv/Gratobpur

Comment author: ciphergoth 20 April 2009 07:09:36AM 20 points [-]

it was said by them that Christopher Hitchens should have watched the theist's earlier debates and been prepared, so I decided not to do that

I urge you to prepare properly. Not only Hitchens but Richard Carrier and several other atheists have been humiliated in debate with him, by their own admission. Winning at all is challenge enough, and would be a great service to the world. Given how much of a blow you would find it to lose having fully prepared, I urge you to to reconsider whether you're self-handicapping.

Comment author: Yvain 20 April 2009 07:22:58AM *  9 points [-]

Who is this theist? I'm interested in watching these debates. (though obviously without knowledge of the specific case, I agree with ciphergoth. It's not just about you, it's about whoever's watching.)

Comment author: Jordan 18 April 2009 06:03:45AM *  0 points [-]

I guess #1. Here are my five:

1) I've climbed Pão de Açucar

2) Every man I've kissed has been dressed like a woman

3) I've never had a single tooth cavity

4) I once hung off the wing of a plane (that was flying)

5) I'm going bald

Comment author: Yvain 19 April 2009 05:22:53PM 0 points [-]

Well, by devoting so much explanation to 2, you make it pretty clear that's not it (or you're just a very good faker.) I guess (.1, .1, .1, .5, .2)

Comment author: [deleted] 18 April 2009 11:16:42AM *  0 points [-]

deleted

In response to comment by [deleted] on Just for fun - let's play a game.
Comment author: Yvain 19 April 2009 05:20:52PM *  0 points [-]

My probabilities are: .1, .3, .2, .2, .2

Comment author: Yvain 19 April 2009 05:20:10PM 9 points [-]

I like this game, but you've got to make it interesting. In that vein, here's mine:

1) Eliezer once wrote an Overcoming Bias post where he said something like "Make one little mistake in a whole long chain of inferences, and you end up in Outer Mongolia." The first time I read this post was from a webcafe in Outer Mongolia.

2) I have pet several full-grown tigers, and one of them almost fell asleep on my lap.

3) I once broke up with a girl partly because she was a deontologist.

4) One of my school teachers once accused me, in all seriousness, of being a witch. Not witch as in "Wiccan", but witch as in "has magic powers and consorts with the devil".

5) I was given a blessing by a Tibetan lama in the mountains of Nepal.

Comment author: Yvain 18 April 2009 01:26:56PM 26 points [-]

If the world were merely seductive, that would be easy. If it were merely challenging, that would be no problem. But I arise in the morning torn between a desire to improve the world, and a desire to enjoy the world. This makes it hard to plan the day.

-- E. B. White

Comment author: Yvain 18 April 2009 12:57:56PM *  10 points [-]

Nothing defines humans better than their willingness to do irrational things in the pursuit of phenomenally unlikely payoffs. This is the principle behind lotteries, dating, and religion.

-- Scott Adams

Comment author: steven0461 17 April 2009 10:04:36PM 2 points [-]

So let's call them hedon-seconds instead.

Comment author: Yvain 17 April 2009 10:29:34PM 1 point [-]

The terminology here is kind of catching me in between a rock and a hard place.

My entire point is that the "utility" of "utilitarianism" might need more complexity than the "utility" of economics, because if someone thinks they prefer a new toaster but they actually wouldn't be any happier with it, I don't place any importance on getting them a new toaster. IANAEBAFAIK economists' utility either would get them the new toaster or doesn't really consider this problem.

...but I also am afraid of straight out saying "Happiness!", because if you do that you're vulnerable to wireheading. Especially with a word like "hedon" which sounds like "hedonism", which is very different from the "happiness" I want to talk about.

CEV might help here, but I do need to think about it more.

Comment author: Yvain 17 April 2009 09:24:47PM *  3 points [-]
  1. It seems that it is possible to compare happiness of two different people; ie I can say that giving the cake to Mary would give her twice as much happiness as it would give Fred. I think that's all you need to counter your first objection. You'd need something much more formal if you were actually trying to calculate it out rather than use it as a principle, but as far as I know no one does this.

  2. This is a big problem. I personally solve it by not using utilitarianism on situations that create or remove people. This is an inelegant hack, but it works.

  3. This is why I said I am a descriptive emotivist but a normative utilitarian. The fact that people don't act in accordance with a system doesn't mean the system isn't moral. I'd be pretty dubious of any moral system that said people were actually doing everything right.

  4. Yeah, tell me about it. Right now I'm thinking that a perfectly rational person has no essential discounts, but ends up with a very hefty discount because she can't make future plans with high effectiveness. For example, investing all my money now and donating the sum+interest to charity in a thousand years only works if I'm sure both the banking system and human suffering will last a millennium.

"Utilons don't make people happier" is a weird way of putting things. It sounds to me a lot like "meters don't make something longer." If you're adding meters to something, and it's not getting longer, you're using the word "meter" wrong.

I don't know much about academic consequentialism, but I'd be surprised if someone hadn't come up with the idea of the utilon x second, ie adding a time dimension and trying to maximize utilon x seconds. If giving someone a new car only makes them happier for the first few weeks, then that only provides so many utilon x seconds. If getting married makes you happier for the rest of your life, well, that provides more utilon x seconds. If you want to know whether you should invest your effort in getting people more cars or getting them into relationships, you'll want to take that into account.

Probably an intelligent theory of utilon x seconds would end up looking completely different from modern consumer culture. Probably anyone who applied it would also be much much happier than a modern consumer. If people can't calculate what does and doesn't provide them with utilon x seconds, they either need to learn to do so, ask someone who has learned to do so to help manage their life, or resign themselves to being less than maximally happy.

I have a feeling this is very different from the way economists think about utility, but that's not necessarily a bad thing.

In response to My Way
Comment author: Yvain 17 April 2009 03:38:52AM 9 points [-]

Without getting into the meat of your post: this is the second or third time you've recommended Jacqueline Carey. The first time you recommended her, I checked out the Kushiel series because I've tended to agree with your taste about some other things. I don't think I ever thanked you for the recommendation, but I need to. Thank you.

Everyone else: get and read the Kushiel series. You won't regret it.

Comment author: Alicorn 17 April 2009 02:41:04AM *  10 points [-]

I'm taking an entire course called "Weird Forms of Consequentialism", so please clarify - when you say "utilitarianism", do you speak here of direct, actual-consequence, evaluative, hedonic, maximizing, aggregative, total, universal, equal, agent-neutral consequentialism?

Comment author: Yvain 17 April 2009 03:34:12AM *  7 points [-]

Uh.....er....maybe!

I'm familiar with Bentham, Mill, Singer, Eliezer, and random snippets of utilitarian theory I picked up here and there. I'm not confident enough with my taxonomy to use quite so many adjectives with confidence. I will add that article to the list of things to read.

I agree that your course sounds awesome. If you hear anything particularly enlightening, please turn it into an LW post.

Comment author: pjeby 17 April 2009 02:41:09AM 16 points [-]

Then the person asks why we think it's bad, and our unconscious supplies whatever rationale it thinks is most plausible and feeds it to us.

Don't blame the unconscious. It only makes up explanations when you ask for them.

My first lesson in this was when I was 17 years old, at my first programming job in the USA. I hadn't been working there very long, maybe only a week or two, and I said something or other that I hadn't thought through -- essentially making up an explanation.

The boss reprimanded me, and told me of something he called "Counter man syndrome", wherein a person behind a counter comes to believe that they know things they don't know, because, after all, they're the person behind the counter. So they can't just answer a question with "I don't know"... and thus they make something up, without really paying attention to the fact that they're making it up. Pretty soon, they don't know the difference between the facts and their own bullshit.

From then on, I never believed my own made-up explanations... at least not in the field of computers. Instead, I considered them as hypotheses.

So, it's not only a learnable skill, it can be learned quickly, at least by 17 year-old. ;-)

Comment author: Yvain 17 April 2009 03:31:28AM 0 points [-]

I think we're talking about subtly different things here. You're talking about explanations of external events, I'm talking about explanations for your own mind states, ie why am I sad right now.

I don't like blaming the "unconscious" or even using the word - it sounds too Freudian - but there aren't any other good terms that mean the same thing.

Comment author: AnnaSalamon 17 April 2009 03:19:10AM 2 points [-]

Yvain, I'm slightly off-topic here, but did you ever do anything with your LW survey? I'm curious to see peoples' responses.

Comment author: Yvain 17 April 2009 03:23:21AM 1 point [-]

I thought that your survey was going to be the same sort of thing, so I didn't develop mine further. Now that I see your survey is after something different, I'll probably try mine again.

Comment author: thomblake 17 April 2009 02:14:55AM *  3 points [-]

Emotivism has its problems. Notably, you can't use 'yay' and 'boo' exclamations in arguments, and they can't be reasons.

"Should I eat this apple?" Becomes simply "how do I feel about eating this apple" (or otherwise it's simply meaningless). But really there are considerations that go into the answer other than mere feelings (for example, is the apple poisonous?).

Because utilitarianism has a theory of right action and a theory of value, I don't think it's compatible with emotivism. But I haven't read much in the literature detailing this particular question, as I don't read much currently about utilitarianism.

Comment author: Yvain 17 April 2009 02:25:08AM *  9 points [-]

Well, what's interesting about that comment is that our beliefs about our own justifications and actions are usually educated guesses and not privileged knowledge. Or consider Eliezer's post about the guy who said he didn't respect Eliezer's ideas because Eliezer didn't have a Ph.D, and then when Eliezer found a Ph.D who agreed with him, the guy didn't believe him either.

My guess would be that we see the apple is poisonous and "downvote" it heavily. Then someone asks what we think of the apple, we note the downvotes, and we say it's bad. Then the person asks why we think it's bad, and our unconscious supplies whatever rationale it thinks is most plausible and feeds it to us. Which is probably that it's poisonous.

See also: footnote 1

Comment author: Yvain 17 April 2009 02:10:24AM 0 points [-]

While we are on the subject of advertisements for things on decision making we technically haven't read: Jonah Lehrer's How We Decide.

I haven't read the book (I just got it today), but Lehrer's blog, Frontal Cortex is one of the top psych/neuro blogs out there and his interest in decision-making runs along pretty much the same track as ours.

The Trouble With "Good"

83 Post author: Yvain 17 April 2009 02:07AM

Related to: How An Algorithm Feels From Inside, The Affect Heuristic, The Power of Positivist Thinking

I am a normative utilitarian and a descriptive emotivist: I believe utilitarianism is the correct way to resolve moral problems, but that the normal mental algorithms for resolving moral problems use emotivism.

Emotivism, aka the yay/boo theory, is the belief that moral statements, however official they may sound, are merely personal opinions of preference or dislike. Thus, "feeding the hungry is a moral duty" corresponds to "yay for feeding the hungry!" and "murdering kittens is wrong" corresponds to "boo for kitten murderers!"

Emotivism is a very nice theory of what people actually mean when they make moral statements. Billions of people around the world, even the non-religious, happily make moral statements every day without having any idea what they reduce to or feeling like they ought to reduce to anything.

Emotivism also does a remarkably good job capturing the common meanings of the words "good" and "bad". An average person may have beliefs like "pizza is good, but seafood is bad", "Israel is good, but Palestine is bad", "the book was good, but the movie was bad", "atheism is good, theism is bad", "evolution is good, creationism is bad", and "dogs are good, but cats are bad". Some of these seem to be moral beliefs, others seem to be factual beliefs, and others seem to be personal preferences. But we are happy using the word "good" for all of them, and it doesn't feel like we're using the same word in several different ways, the way it does when we use "right" to mean both "correct" and "opposite of left". It feels like they're all just the same thing. The moral theory that captures that feeling is emotivism. Yay pizza, books, Israelis, atheists, dogs, and evolution! Boo seafood, Palestinians, movies, theists, creationism, and cats!

Remember, evolution is a crazy tinker who recycles everything. So it would not be surprising to find that our morality is a quick hack on the same machinery that runs our decisions about which food to eat or which pet to adopt. To make an outrageous metaphor: our brains run a system rather like Less Wrong's karma. You're allergic to cats, so you down-vote "cats" a couple of points. You hear about a Palestinian committing a terrorist attack, so you down-vote "Palestinians" a few points. Richard Dawkins just said something especially witty, so you up-vote "atheism". High karma score means seek it, use it, acquire it, or endorse it. Low karma score means avoid it, ignore it, discard it, or condemn it.1

Remember back during the presidential election, when a McCain supporter claimed that an Obama supporter attacked her and carved a "B" on her face with a knife? This was HUGE news. All of my Republican friends started emailing me  and saying "Hey, did you hear about this, this proves we've been right all along!" And all my Democratic friends were grumbling and saying how it was probably made up and how we should all just forget the whole thing.

And then it turned out it WAS all made up, and the McCain supporter had faked the whole affair. And now all of my Democrat friends started emailing me and saying "Hey, did you hear about this, it shows what those Republicans and McCain supporters are REALLY like!" and so on, and the Republicans were trying to bury it as quickly as possible.

The overwhelmingly interesting thing I noticed here was that everyone seemed to accept - not explicitly, but implicitly very much - that an Obama supporter acting violently was in some sense evidence against Obama or justification for opposition to Obama; or, that a McCain supporter acting dishonestly was in some sense evidence against McCain or confirmation that Obama supporters were better people. To a Bayesian, this would be balderdash. But to an emotivist, where any bad feelings associated with Obama count against him, it sort of makes sense. All those people emailing me about this were saying: Look, here is something negative associated with Obama; downvote him!2

So this is one problem: the inputs to our mental karma system aren't always closely related to the real merit of a person/thing/idea.

Another problem: our interpretation of whether to upvote or downvote something depends on how many upvotes or downvotes it already has. Here on Less Wrong we call this an information cascade. In the mind, we call it an Affective Death Spiral.

Another problem: we are tempted to assign everything about a concept the same score. Eliezer Yudkowsky currently has 2486 karma. How good is Eliezer at philosophy? Apparently somewhere around the level it would take to get 2486 karma. How much does he know about economics? Somewhere around level 2486 would be my guess. How well does he write? Probably well enough to get 2486 karma. Translated into mental terms, this looks like the Halo Effect. Yes, we can pick apart our analyses in greater detail; having read Eliezer's posts, I know he's better at some things than others. But that 2486 number is going to cause anchoring-and-adjustment issues even so.

But the big problem, the world-breaking problem, is that sticking everything good and bad about something into one big bin and making decisions based on whether it's a net positive or a net negative is an unsubtle, leaky heuristic completely unsuitable for complicated problems.

Take gun control. Are guns good or bad? My gut-level emotivist response is: bad. They're loud and scary and dangerous and they shoot people and often kill them. It is very tempting to say: guns are bad, therefore we should have fewer of them, therefore gun control. I'm not saying gun control is therefore wrong: reversed stupidity is not intelligence. I'm just saying that before you can rationally consider whether or not gun control is wrong, you need to get past this mode of thinking about the problem.

In the hopes of using theism less often, a bunch of Less Wrongers have agreed that the War on Drugs would make a good stock example of irrationality. So, why is the War on Drugs so popular? I think it's because drugs are obviously BAD. They addict people, break up their families, destroy their health, drive them into poverty, and eventually kill them. If we've got to have a category "drugs"3, and we've got to call it either "good" or "bad", then "bad" is clearly the way to go. And if drugs are bad, getting rid of them would be good! Right?

So how do we avoid all of these problems?

I said at the very beginning that I think we should switch to solving moral problems through utilitarianism. But we can't do that directly. If we ask utilitarianism "Are drugs good or bad?" it returns: CATEGORY ERROR. Good for it.

Utilitarianism can only be applied to states, actions, or decisions, and it can only return a comparative result. Want to know whether stopping or diverting the trolley in the Trolley Problem would be better? Utilitarianism can tell you. That's because it's a decision between two alternatives (alternate way of looking at it: two possible actions; or two possible states) and all you need to do is figure out which of the two is higher utility.

When people say "Utilitarianism says slavery is bad" or "Utilitarianism says murder is wrong" - well, a utilitarian would endorse those statements over their opposites, but it takes a lot of interpretation first. What utilitarianism properly says is "In this particular situation, the action of freeing the slaves leads to a higher utility state than not doing so" and possibly "and the same would be true of any broadly similar situation".

But why in blue blazes can't we just go ahead and say "slavery is bad"? What could possibly go wrong?

Ask an anarchist. Taxation of X% means you're forced to work for X% of the year without getting paid. Therefore, since slavery is "being forced to work without pay" taxation is slavery. Since slavery is bad, taxation is bad. Therefore government is bad and statists are no better than slavemasters.4

(again, reversed stupidity is not intelligence. There are good arguments against taxation. But this is not one of them.)

Emotivism is the native architecture of the human mind. No one can think like a utilitarian all the time. But when you are in an Irresolvable Debate, utilitarian thinking may become necessary to avoid dangling variable problems around the word "good" (cf. Islam is a religion of peace). Problems that are insoluble at the emotivist level can be reduced, simplified, and resolved on the utilitarian level with enough effort.

I've used the example before, and I'll use it again. Israel versus Palestine. One person can go on and on for months about all the reasons the Israelis are totally right and the Palestinians are completely in the wrong, and another person can go on just as long about how the Israelis are evil oppressors and the Palestinians just want freedom. And then if you ask them about an action, or a decision, or a state - they've never thought about it. They'll both answer something like "I dunno, the two-state solution or something?". And if they still disagree at this level, you can suddenly apply the full power of utilitarianism to the problem in a way that tugs sideways to all of their personal prejudices.

In general, any debate about whether something is "good" or "bad" is sketchy, and can be changed to a more useful form by converting the thing to an action and applying utilitarianism.

Footnotes:

1: It should be noted that this karma analogy can't explain our original perception of good and bad, only the system we use for combining, processing and utilizing it. My guess is that the original judgment of good or bad takes place through association with other previously determined good or bad things, down to the bottom level which are programmed into the organism (ie pain, hunger, death) with some input from the rational centers.

2: More evidence: we tend to like the idea of "good" or "bad" being innate qualities of objects. Thus the alternative medicine practioner who tells you that real medicine is bad, because it uses scary pungent chemicals, which are unhealthy, and alternative medicine is good, because it uses roots and plants and flowers, which everyone likes. Or fantasy books, where the Golden Sword of Holy Light can only be wielded for good, and the Dark Sword of Demonic Shadow can only be wielded for evil.

3: Of course, the battle has already been half-lost once you have a category "drugs". Eliezer once mentioned something about how considering {Adolf Hitler, Joe Stalin, John Smith} a natural category isn't going to do John Smith any good, no matter how nice a man he may be. In the category "drugs", which looks like {cocaine, heroin, LSD, marijuana}, LSD and marijuana get to play the role of John Smith.

4: And, uh, I'm sure Louis XVI would feel the same way. Sorry. I couldn't think of a better example.

Comment author: Yvain 17 April 2009 01:31:20AM *  2 points [-]

Good survey, and I like the calibration questions: coming up with good ones is tough. But I suggest the following:

Create an "as happy as other people" for the happiness questions, instead of just "less" or "more"

Define "auto accident" better (I had one that sort of nicked my bumper)

Define "exercise" better (I walk everywhere I go, totalling at least an hour per day, but I don't go to gyms. Does that count?)

In "have you earned more or less than those you grew up with?", include an option for "I am in school" - I think a lot of us are earning less than our peers, but only because we're in some kind of graduate education while they're out making money

Comment author: Yvain 15 April 2009 07:01:33PM 37 points [-]

IAWYC, but I think it sidesteps an important issue.

A perfectly rational community will be able to resist the barbarians. But it's possible, perhaps likely, that as you increase community rationality, there's a valley somewhere between barbarian and Bayesian where fighting ability decreases until you climb out of it.

I think the most rational societies currently existing are still within that valley. And that a country with the values and rationality level of 21st century Harvard will with high probability be defeated by a country with the values and rationality level of 13th century Mongolia (holding everything else equal).

I don't know who you're arguing against, but I bet they are more interested in this problem than in an ideal case with a country of perfect Bayesians.

Comment author: Yvain 14 April 2009 09:33:36PM *  29 points [-]

Blink. You read Reddit, right? Have you never noticed that every time there's an outrageous story, everyone on Reddit bands together and does something about it? Dusty the cat? The ReMax debacle? That woman who got her cruise cancelled and the Redditors sent enough to get her a new one? Also, http://www.cracked.com/article_17170_8-awesome-cases-internet-vigilantism.html . This is pretty impressive. If I'd, say, put a big poster up in a school about Dusty the Cat or ReMax, I doubt the students would have been able to mount half as coherent or overwhelming a response as the Internet did.

And Anonymous versus Scientology was pretty impressive too.

All of these have some things in common. They're responses to a single outrageous incident, they're things that the mainstream media doesn't cover, and they don't take a huge time commitment to solve. So there is a big difference between them and, say, fighting world hunger.

But what I gather from these examples is that anonymity and the bystander effect do not suddenly change the incentive structure for people online. Possibly the best known Internet action-taking campaign ever was the anti-Scientology one perpetrated by...Anonymous.

I would suggest we shift our inquiries in the direction of why the Internet is so good at Dusty the Cat style operations and so bad at end world hunger style operations. I think it probably has to do with the way people use the Internet itself: short attention spans and novelty-seeking.

On the other hand, the Internet can pull through for people long-term: witness Howard Dean, Barack Obama, Ron Paul, and "netroots". So maybe it has more to do with the fragmented nature of the Internet. Reddit is a natural place for Ron Paul fans to get together and organize Ron Paul related things, but there are lots of fragmented communities and none of them is specifically focused on world hunger. Nor would a sudden interest in solving world hunger on one community's part spread to another.

I don't know. Don't have a specific answer. Just think we need to shift direction away from "Why is the Internet so bad at this?" because it isn't

Comment author: Yvain 14 April 2009 09:14:43PM 0 points [-]

Interesting.

I don't really think 2 is a deliberate strategy as much as the way things turn out. Richard Dawkins writes a book on atheism because he thinks people need to know about it and society should talk about it more and so on. The only people who buy it are the atheists, because they want to signal their atheism and the Christians don't want to read something they disagree with any more than we'd pick up a book on creationism.

It sounds like in your first paragraph about debates, you're saying a lot of people target their arguments ineffectively. Any theories on why that is, or whether there are specific biases involved?

In response to How Much Thought
Comment author: JulianMorrison 12 April 2009 11:58:39AM *  12 points [-]

Serious question: how do people on LW experience thinking?

I find that the heavy lifting of my thinking is nonverbal and fast. Verbalizing (and often re-re-verbalizing) takes time, running explicit math takes (a lot of) time, and worrying at a verbalized idea without getting anywhere can waste indefinite time, but actual intelligent thought feels like a burst of focused attention/effort and a gestalt answer that arrives in an instant.

Comment author: Yvain 12 April 2009 03:31:26PM *  7 points [-]

I get a vague strained feeling to which I don't have full conscious access, which transforms into a form that makes sense to me only after I've verbalized it.

Comment author: Eliezer_Yudkowsky 10 April 2009 11:32:41PM 12 points [-]

In a world where everyone is sort of a jerk and says "Just shut up and exercise, you fatso!" there may be such a strong drive to avoid condemnation and low social status that you actually do shut up and exercise.

I did shut up and exercise. It didn't work. That's the point at which you have a problem.

And for years I felt guilty and that I must be doing something wrong; and then I read about the Shangri-La diet and all these people losing 50 pounds with ease; and then it didn't work for me; and that was when I figured out that yes, I actually had put in a really serious try, and that what was really going on was that the laws just didn't work the good and virtuous and just way that everyone said they did.

Now maybe for other things... if willpower really does work... then telling people "Shut up and expend willpower" might be helpful. I've just gotten a lot more skeptical, now.

Comment author: Yvain 10 April 2009 11:55:21PM *  2 points [-]

Wasn't talking about your case in particular. Of would-be-dieters I know, the majority try to go to the gym a few times and then flake out. So although it may apply to you, I don't think "You can't just tell people to try harder" is always good advice.

Comment author: Yvain 10 April 2009 11:43:28PM *  11 points [-]

"There are no outs. Even if someone else would call it an extenuating circumstance and forgive me for giving up, I'll just get it done anyway."

This post and Extenuating Circumstances aren't literally contradictory, but their implications seem to point in opposite directions. I would like to see more discussion of when to apply this mode of thinking and when to apply the Extenuating Circumstances mode of thinking.

Right now I'm interpreting the difference as being that if you really want to lose weight, you shouldn't accept "I have an inconvenient metabolic set point" as an excuse not to do so, but you should realize that it will shift which routes are easier than others and take that into account when planning your best strategy for weight loss. So you might try devote effort to finding some clever trick instead of trying to steamroller ahead with sheer willpower.

Am I on the right track?

Comment author: jimrandomh 10 April 2009 09:29:12PM 3 points [-]

I have to second Eliezer on this one. Saying "good diet and exercise" is just a disguised way of saying "be more disciplined". While it is true that being more disciplined would cause someone to lose weight, telling someone to be more disciplined does not cause them to actually /be/ more disciplined. The value of advice is properly judged by its effect, and actual observation shows that the "be more disciplined" advice has no effect or even the opposite effect, so it's simply bad advice. The part which is true is already known by the person receiving the advice, so truth is no defense.

Comment author: Yvain 10 April 2009 11:26:04PM 11 points [-]

This touches on a general issue about free will.

In a world where everyone is sort of a jerk and says "Just shut up and exercise, you fatso!" there may be such a strong drive to avoid condemnation and low social status that you actually do shut up and exercise.

In the alternate world where everyone understands that it's not really your fault and you can't shout people into having more willpower and willpower is a sketchy concept anyway and accepts you for who you are - you will have no incentive to get better.

So occasionally I do tell people the equivalent of "shut up and exercise" for certain things, even though I know it doesn't work directly. It's a case by case basis, depending on how many opportunities the person is missing and how likely I think my advice is to seriously affect them.

Comment author: Yvain 10 April 2009 08:37:09PM *  4 points [-]

I hope you didn't interpret my post Wednesday as saying that nothing you wrote was useful. My only gripe was that people seemed to be talking in terms of "this is absolutely certain to change the world and transform us all into ubermenschen!" and that we should start off more sober. Or maybe no one was really talking that way, and I was misinterpreting people's deliberate hyperbole in terms of my own Happy Death Spiral. But that was all I was arguing against. Thus the admission that rationality should have a .1 correlation with success, and the comment that "Good use of rationality will look more like three percent productivity gain than Napoleon conquering Europe".

I think Crisis of Faith can be good for certain situations, but I am skeptical about it being completely game-changing for a few reasons.

Most smart people already have a naive version of this technique: that when all the evidence is going against them, they need to stop and think about whether their beliefs are right or wrong. For Crisis of Faith to be practically valuable, you need lots of cases where:

(1) EITHER people don't apply the naive technique often enough in situations where it could give practical real-world benefits, and formalizing it will convince them to do it more often,
(2) OR the specific advice you give in the Crisis of Faith post makes a full-blown Crisis of Faith more likely to return the correct answer than the naive technique.
(3) AND once they finish Crisis of Faith, they go through with their decision.

I give (1) low probability. People don't change their religious or political views often enough, but they're often good at changing their practical situations. I've heard many people tell stories of how they stayed up all night agonizing over whether or not to break up with a girlfriend. In many cases I think the difficulty is in reaching the point where you admit "I need to seriously reconsider this." I doubt many people reach a point where they feel uncomfortable about their position on a practical issue but don't take any time to think it over. And the people who would be interested in rationalism are probably exactly the sort of people who currently use the naive technique most often already. I used a naive version of Crisis of Faith for a very important decision in my life before reading OB.

I give (2) high but not overwhelming probability. Yes, all of this ought to work. But I was reading up on evidence-based medicine last night in response to your comment, and one thing that struck me the most was that "ought to work" is a very suspicious phrase. Doctors possessing mountains of accurate information about liver function can say "From what we know about the liver, it would be absolutely absurd for this chemical not to cure liver disease" and then they do a study and it doesn't help at all. With our current state of knowledge and the complexity of the subject, it's easier to make mistakes about rationality than about the liver. Yes, my completely non-empirical gut feeling is that the specific components of Crisis of Faith should work better than just sitting and thinking, but maybe anyone unbiased enough to succeed at Crisis of Faith is unbiased enough to succeed at the naive method. Maybe Crisis of Faith creates too much pressure to reject your old belief in order to signal rationality, even if the old belief was correct.

I give (3) medium probability.

And all this is an upper bound anyway. The next question is whether some specific training program will teach people to use Crisis of Faith regularly and correctly. I predict that the "training program" of reading about it on Overcoming Bias in most cases does not, but this is easy to test:

Everyone please comment below if (how many times?) you've actually used the formal Crisis of Faith technique in a situation where you wouldn't have questioned something if you hadn't read the Crisis of Faith article. Please also mention whether it was about a practical real-world matter, and whether you ended up changing your mind

Again, not saying this to prove Crisis of Faith is worthless, just to show that there are factors to be considered beyond its raw value as a technique.

Comment author: ciphergoth 10 April 2009 08:21:19AM 5 points [-]

The so-called terrorism threat?

I still have that I must be missing something feeling about that one. It seems so obvious that terrorism is very low down on the chart of public health problems, and that little would do more to defeat their aims than treating it as such; but when someone like Hitchens takes it seriously, I really want to know if there's an argument I haven't considered.

Comment author: Yvain 10 April 2009 02:35:25PM 5 points [-]

One of my smarter friends supports the War on Terror. His rationale, which I find at least worth consideration, is that left alone, there's a decent chance terrorists will steal a nuclear weapon from Pakistan or the former USSR or somewhere and nuke a major city. Considering how much damage that could do, and how not-really-unlikely this is, the expected damage really is greater than that from a lot of other problems. Although we should be putting extra resources into fighting this specific problem (like securing nukes better) a large part of the strategy has to be an all-out war on terror.

I'm pretty sure he doesn't think the war on Iraq is very helpful, but he thinks that wiretaps, detention camps, the war in Afghanistan, and vastly increased border security are all part of this effort.

Comment author: Yvain 10 April 2009 03:16:53AM 5 points [-]

In the spirit of concrete reductions, I have a question for everyone here:

Let's say we took a random but very large sample of students from prestigious colleges, split them into two groups, and made Group A take a year-long class based on Overcoming Bias, in which students read the posts and then (intelligent, engaging) professors explained anything the students didn't understand. Wherever a specific technique was mentioned, students were asked to try that technique as homework.

Group B took a placebo statistics class similar to every other college statistics class, also with intelligent and engaging professors.

Twenty-five years later, how would you expect the salaries of students in Group A to compare to the salaries of students in Group B? The same? 1.1 times greater? Twice as great? What about self-reported happiness? Amount of money donated to charity per year?

Comment author: Yvain 10 April 2009 03:05:17AM 4 points [-]

This is a great post.

Comment author: Eliezer_Yudkowsky 09 April 2009 09:10:52PM 13 points [-]

One thinks particularly of Robyn Dawes - I don't know him from "evidence-based medicine" per se, but I know he was fighting the battle to get doctors to acknowledge that their "clinical experience" wasn't better than simple linear models, and he was on the front lines against psychotherapy shown to perform no better than talking to any bright person.

If you read "Rational Choice in an Uncertain World" you will see that Dawes is pretty definitely on the level of "integrate Bayes into everyday life", not just Traditional Rationality. I don't know about the historical origins of evidence-based medicine, so it's possible that a bunch of Traditional Rationalists invented it; but one does get the impression that probability theorists trying to get people to listen to the research about the limits of their own minds, were involved.

Comment author: Yvain 10 April 2009 02:59:04AM 23 points [-]

After thinking on this for a while, here are my thoughts. This should probably be a new post but I don't want to start another whole chain of discussions on this issue.

  1. I had the belief that many people on Less Wrong believed that our currently existing Art of Rationality was sufficient or close to sufficient to guarantee practical success or even to transform its practioner into an ubermensch like John Galt. I'm no longer sure anyone believes this. If they do, they are wrong. If anyone right now claims they participate in Less Wrong solely out of a calculated program to maximize practical benefits and not because they like rationality, I think they are deluded.

  2. Where x-rationality is defined as "formal, math-based rationality", there are many cases of x-rationality being used for good practical effect. I missed these because they look more like three percent annual gains in productivity than like Brennan discovering quantum gravity or Napoleon conquering Europe. For example, doctors can use evidence-based medicine to increase their cure rate.

  3. The doctors who invented evidence-based medicine deserve our praise. Eliezer is willing to consider them x-rationalists. But there is no evidence that they took a particularly philosophical view towards rationality, as opposed to just thinking "Hey, if we apply these tests, it will improve medicine a bit." Depending on your view of socialism, the information that one of these inventors ran for parliament on a socialist platform may be an interesting data point.

  4. These doctors probably had mastery of statistics, good understanding of the power of the experimental method, and a belief that formalizing things could do better than normal human expertise. All of these are rationalist virtues. Any new doctor who starts their career with these virtues will be in a better position to profit from and maybe expand upon evidence-based medicine than a less virtuous doctor, and will reap great benefits from their virtues. Insofar as Less Wrong's goal is to teach people to become such doctors, this is great...

  5. ...except that epidemiology and statistics classes teach the same thing with a lot less fuss. Less Wrong's goal seems to be much higher. Less Wrong wants a doctor who can do that, and understand their mental processes in great detail, and who will be able to think rationally about politics and religion and turn the whole thing into a unified rationalist outlook.

  6. Or maybe it doesn't. Eliezer has already explained that a lot of his OB writing was just stuff that he came across trying to solve AI problems. Maybe this has turned us into a community of people who like talking about philosophy, and that really doesn't matter much and shouldn't be taught at rationality dojos. Maybe a rationality dojo should be an extra-well-taught applied statistics class and some discussion of important cognitive biases and how to avoid them. It seems to me that a statistics class plus some discussion of cognitive biases would be enough to transform an average doctor into the kind of doctor who could invent or at least use evidence-based medicine and whatever other x-rationality techniques might be useful in medicine. With a few modifications, the same goes for business, science, and any other practical field.

  7. I predict the marginal utility of this sort of rationality will decline quickly. The first year of training will probably do wonders. The second year will be less impressive. I doubt a doctor who studies this rationality for ten years will be noticeably better off than one who studies it for five, although this may be my pessimism speaking. Probably the doctor would be better off spending those second five years studying some other area of medicine. In the end, I predict these kinds of classes could improve performance in some fields 10-20% for people who really understood them.

  8. This would be a useful service, but it wouldn't have the same kind of awesomeness as Overcoming Bias did. There seems to be a second movement afoot here, one to use rationality to radically transform our lives and thought processes, moving so far beyond mere domain-specific reasoning ability that even in areas like religion, politics, morality, and philosophy we hold only rational beliefs and are completely inhospitable to any irrational thoughts. This is a very different sort of task.

  9. This new level of rationality has benefits, but they are less practical. There are mental clarity benefits, and benefits to society when we stop encouraging harmful political and social movements, and benefits to the world when we give charity more efficiently. Once people finish the course mentioned in (6) and start on the course mentioned in (8), it seems less honest to keep telling them about the vast practical benefits they will attain.

  10. This might have certain social benefits, but you would have to be pretty impressive for conscious-level social reasoning to get better than the dedicated unconscious modules we already use for that task.

  11. I have a hard time judging opinion here, but it does seem like some people think sufficient study of z-rationality can turn someone into an ubermensch. But the practical benefits beyond those offered by y-rationality seem low. I really like z-rationality, but only because I think it's philosophically interesting and can improve society, not because I think it can help me personally.

  12. In the original post, I was using x-rationality in a confused way, but I think to some degree I was thinking of (8) rather than (6).

Comment author: Yvain 09 April 2009 11:44:04PM 12 points [-]

This idea has internal relevance as well. You could easily consider, for instance, the self introspecting at home who wants to lose weight and the self in a restaurant who wants to order cheesecake as two sides of a Silver Chair**. And I think that view is more helpful than just calling it "akrasia", because it presents the situation as two aspects of your personality which happen to want different things, instead of some "weakness" which is interfering with your "true will". Then instead of castigating yourself for weakness of will, you merely think "I suppose my desire for cheesecake was stronger than I anticipated. When I return to a state where my desire to lose weight is dominant, I shall have to make stricter plans."

IAWYC, but I have trouble with this particular example. Quite often when I do eat that piece of cheesecake, I'm thinking "Oh no, I hate myself, I really shouldn't be doing this". On the other hand, I have no such feelings when the diet-self wins out over the dessert-self. That suggests that there is some fundamental asymmetry, and not just two different but equal selves involved.

Comment author: cousin_it 09 April 2009 09:28:56PM *  2 points [-]

Your post is a great improvement on mine. Thanks, esp. for the "limiting factor" riff.

Am I alone in thinking the word "akrasia" doesn't quite describe our problem? Isn't it more like "apathy"? Some people wish to be able to do the things they want; lucky them! Me, I just wish to want to do the things I'm able to do.

Comment author: Yvain 09 April 2009 10:37:37PM 0 points [-]

You're welcome, even though I was pretty sure I was arguing against you. My poor models of other people's opinions strike again.

Comment author: Eliezer_Yudkowsky 09 April 2009 02:18:26PM 23 points [-]

An x-rationalist who becomes a doctor would not, I think, necessarily be a significantly better doctor than the rest of the medical world, because the rest of the medical world already has an overabundance of great rationality techniques and methods of improving care that the majority of doctors just don't use

Evidence-based medicine was developed by x-rationalists. And to this day, many doctors ignore it because they are not x-rationalists.

Comment author: Yvain 09 April 2009 09:00:01PM 10 points [-]

...huh. That comment was probably more helpful than you expected it to be. I'm pretty sure I've identified part of my problem as having too high a standard for what makes an x-rationalist. If you let the doctors who developed evidence-based medicine in...yes, that clears a few things up.

Comment author: AnnaSalamon 09 April 2009 01:10:20PM *  8 points [-]

This experiment seems easy to rig4; merely doing it should increase your level of conscious rational decisions quite a bit. And yet I have been trying it for the past few days, and the results have not been pretty. .... [O]ne way to fail your Art is to expect more of it than it can deliver.... Perhaps there are developments of the Art of Rationality or its associated Arts that can turn us into a Kellhus or a Galt, but they will not be reached by trying to overcome biases really really hard.

To make a somewhat uncharitable paraphrase: you read many articles on rationality, did not actually use them to change the way you make decisions, and found that the rationality hasn’t changed the results of your decisions much. You conclude, not that you aren’t practicing rationality, but that rationality can’t deliver practical goods at all, at least not as taught.

I agree we need practices for better incorporating OB/LW/new techniques of rationality into our actual practice of inference and decision-making. But it seems like the “and I’m not actually using this stuff much” result of your experiment should prevent “it hasn’t made my life much better” from telling you all that much about whether the OB/LW inference or decision-making techniques, if one does practice them, could make one’s life better.

Comment author: Yvain 09 April 2009 08:56:26PM *  38 points [-]

I accept that to some degree my results say more negative things about me than about rationality, but insofar as I may be typical we need to take them into account when considering how we're going to benefit from rationality.

...my inability to communicate clearly continues to be the bane of my existence. Let me try a strained metaphor.

Christianity demands its adherents "love thy enemy", "turn the other cheek", "judge not lest ye be judged", "give everything to the poor", and follow many other pieces of excellent moral advice. Any society that actually followed them all would be a very nice place to live.

Yet real-world Christian societies are not such nice places to live. And Christians say this is not because there is anything wrong with Christianity, but because Christians don't follow their religion enough. As the old saying goes, "Christianity has not been tried and found wanting, it has been found difficult and left untried." There's some truth to this.

But it doesn't excuse Christianity's failure to make people especially moral. If Christianity as it really exists can't translate its ideals into action, then it's gone wrong somewhere. At some point you have to go from "Christianity is perfect but most people can't apply it" to "Christianity is flawed because most people can't apply it."

The Christians' problem isn't that there aren't enough Christians. And it's not that Christians aren't devout and zealous enough. And it's not even that Christians don't understand what their faith expects of them. Their problem is that the impulse to love thy neighbor gets lost somewhere between theory and action. My urge as an outsider is to blame it on a combination of akrasia, lack of sufficient self-consciousness, and people who accept Christianity 100% on the conscious level but don't "feel it in their bones".

If I were a theologian, I would be recommending to my fellow Christians one of two things:

First, that they spend a whole lot less time in Bible study than they do right now, because they already know a whole lot more Bible than they actually use, and teaching them more Bible isn't going to solve that problem. Instead they need to be spending that time thinking of ways to solve their problem with applying the Bible to real life.

Or second, that they stop talking about how moral Christianity makes them and how a Christian society will always be a moral society and so on and so it's beneficial that everyone learn Christianity, and just admit that Christians probably aren't that much more moral than atheists and that they're in it because they like religion. In that case they could go on talking about the Bible to their hearts' content.

Now, to some degree, we can blame individual Christians for the failure of Christianity to transform morality for the better. But we also have to wonder if maybe it's not even addressing the real problem, which is less of a lack of moral ideals than a basic human inability to translate moral ideals into action.

Right now I find myself in the same situation as a devout Christian who really wants to be good, but has noticed that studying lots of Bible verses doesn't help him. Less Wrong has thus far seemed to me like a Bible study group where we all get together and talk about how with all this Bible studying we'll all be frickin saints soon. Eliezer's community-building posts seem like Catholics and Episcopalians arguing on the best way to structure the clergy. It's all very interesting, but...

...but I feel like there is insufficient appreciation that the Art of Knowing the Bible and the Art of Becoming a Saint are two very different arts, that we haven't really begun developing the second, and that religion has a bad track record of generating saints anyway.

Your objection sounds too much like saying that since I'm not a saint yet, I must simply not be applying my Bible study right. Which is in one sense true, but centuries of Christians telling each other that hasn't created any more saints. So people need to either create an Art of Becoming A Saint worthy of the name, or stop insisting that we will soon be able to create saints on demand.

Comment author: Eliezer_Yudkowsky 09 April 2009 12:10:10PM 18 points [-]

The above would be component 1 of my own reply.

Component 2 would be (to say it again) that I developed the particular techniques that are to be found in my essays, in the course of solving my problem. And if you were to try to attack that or a similar problem you would suddenly find many more OB posts to be of immensely greater use and indeed necessity. The Eliezer of 2000 and earlier was not remotely capable of getting his job done.

What you're seeing here is the backwash of techniques that seem like they ought to have some general applicability (e.g. Crisis of Faith) but which are not really a whole developed rationalist art, nor made for the purpose of optimizing everyday life.

Someone faced with the epic Challenge Of Changing Their Mind may use the full-fledged Crisis of Faith technique once that year. How much benefit is this really? That's the question, but I'm not sure the cynical answer is the right one.

What I am hoping to see here is others, having been given a piece of the art, taking that art and extending it to cover their own problems, then coming back and describing what they've learned in a sufficiently general sense (informed by relevant science) that I can actually absorb it. For that which has been developed to address e.g. akrasia outside the rationalist line, I have found myself unable to absorb.

Comment author: Yvain 09 April 2009 01:55:35PM *  13 points [-]

But you're not a good test case to see whether rationality is useful in everyday life. Your job description is to fully understand and then create a rational and moral agent. This is the exceptional case where the fuzzy philosophical benefits of rationality suddenly become practical.

One of the fundamental lessons of Overcoming Bias was "All this stuff philosophers have been debating fruitlessly for centuries actually becomes a whole lot clearer when we consider it in terms of actually designing a mind." This isn't surprising; you're the first person who's really gotten to use Near Mode thought on a problem previously considered only in Far Mode. So you've been thinking "Here's this nice practical stuff about thinking that's completely applicable to my goal of building a thinking machine", and we've been thinking, "Oh, wow, this helps solve all of these complicated philosophical issues we've been worrying about for so long."

But in other fields, the rationality is domain-specific and already exists, albeit without the same thunderbolt of enlightenment and awesomeness. Doctors, for example, have a tremendous literature on evidence and decision-making as they relate to medicine (which is one reason I get so annoyed with Robin sometimes). An x-rationalist who becomes a doctor would not, I think, necessarily be a significantly better doctor than the rest of the medical world, because the rest of the medical world already has an overabundance of great rationality techniques and methods of improving care that the majority of doctors just don't use, and because medicine takes so many skills besides rationality that any minor benefits from the x-rationalist's clearer thinking would get lost in the noise. To make this more concrete: I don't think good doctors are more likely to be atheists than bad doctors, though I do think good AI scientists are more likely to be atheists than bad AI scientists. I think this paragraph about doctors also applies to businessmen, scientists, counselors, et cetera.

When I said that we had a non-trivial difference of opinion on your secret identity post, this was what I meant: that a great x-rationalist might be a mediocre doctor; that maybe if you'd gone into medicine instead of AI you would have been a mediocre doctor and then I wouldn't be "allowed" to respect you for your x-rationality work.

Comment author: badger 09 April 2009 06:00:19AM *  10 points [-]

I'm confused about this article. I agree with most you've said, but I'm not sure the point is exactly. I thought the entire premise of this community was that more is possible, but we're only "less wrong" at the moment. I didn't think there was any promise of results for the current state of the art. Is this post a warning, or am I overlooking this trend?

I agree we shouldn't see x-rationality as practically useful now. You don't rule out rationality becoming the superpower Eliezer portrays in his fiction. That is certainly a long ways off. Boyle's Law and weather prediction is an apt analogy. Just trying harder to apply our current knowledge won't go very far, but there should be some productive avenues.

I think I'd understand your purpose better if you could answer these questions: In your mind, how likely is it that x-rationality could be practically useful in, say, 50 years? What approaches are most likely to get us to a useful practice of rationality? Or is your point that any advances that are made will be radically different from our current lines of investigation?

Just trying to understand.

Comment author: Yvain 09 April 2009 12:55:41PM 1 point [-]

I'll admit I might be attacking a straw man, but if you read the posts linked to on the very top, I think there are at least a few people out there who believe it, or who don't consciously believe it but act as if it's true.

How likely is it that x-rationality could be practically useful in, say, 50 years.

Depends how you reduce "practically useful". Reduce it to "a person randomly assigned to take rationality classes two hours a week plus homework for a year will make on average ten percent more money than a similar person who doesn't", my wild completely unsubstantiated guess is 50% likely. But I'd give similar numbers to other types of self-improvement classes like Carnegie seminars and that sort of thing.

What approaches are most likely to get us to a useful practice of rationality? Or is your point that any advances that are made will be radically different from our current lines of investigation?

If by "useful practice of rationality" you mean the way Eliezer imagines it, I think there should be more focus on applying the rationality we have rather than delving deeper and deeper into the theory, but if I could say more than that, I'd be rich and you'd be paying me outrageous hourly fees to talk about it :)

I do think non-godlike levels of rationality have far more potential to help us in politics than in daily life, but that's a minefield. In terms of easy profits we should focus the movement there, but in terms of remaining cohesive and credible it's not really an option.

Comment author: AnnaSalamon 09 April 2009 07:46:28AM *  11 points [-]

I’m partly echoing badger here, but it’s worth distinguishing between three possible claims:
(1) An “art of rationality” that we do not yet have, but that we could plausibly develop with experimentation, measurements, community, etc., can help people.
(2) The “art of rationality” that one can obtain by reading OB/LW and trying to really apply its contents to one’s life, can help people.
(3) The “art of rationality” that one is likely to accidentally obtain by reading articles about it, e.g. on OB/LW, and seeing what happens to rubs off, can help people.

There are also different notions of “help people” that are worth distinguishing. I’ll share my anticipations for each separately. Yvain or others, tell me where your anticipations match or differ.

Regarding claim (3):
My impression is that even the art of rationality one obtains by reading articles about it for entertainment, does have some positive effects on the accuracy of peoples’ beliefs. A couple people reported leaving their religions. Many of us have probably discarded random political or other opinions that we had due to social signaling or happenstance. Yvain and others report “clarity-of-mind benefits”. I’d give reasonable odds that there’s somewhat more benefit than this -- some unreliable improvement in peoples’ occasional, major, practical decisions, e.g. about which career track to pursue, and some unreliable improvement in peoples’ ability to see past their own rationalizations in interpersonal conflicts -- but (at least with hindsight bias?) probably no improvements in practical skills large enough to show up on Vladimir Nesov’s poll. Does anyone’s anticipations differ, here?

Regarding claim (2):
I’d a priori expect better effects from attempts to really practice rationality, and to integrate its thinking skills into one’s bones, than from enjoying chatting about rationality from time to time. A community that reads articles about skateboarding, and discusses skateboarding, will probably still fall over when they try to skateboard twenty feet unless they’ve also actually spent time on skateboards.

As to the empirical data: who here has in fact practiced (2) (e.g., has tried to integrate x-rationality into their actual practical decision-making, as in Yvain’s experiment/technique, or has used x-rationality to make major life decisions, or has spent time listing out their strengths and weaknesses as a rationalist with specific thinking habits that they really work to integrate in different weeks, or etc.)? This is a real question; I’d love data. Eliezer is an obvious example; Yvain cites the impressiveness of Eliezer’s 2001 writings as counter-evidence (and it is some counter-evidence), but: (1) Eliezer, in 2001, had already spent a lot of time learning rationality (though without the heuristics and biases literature); and (2) Eliezer was at that time busy with a course of action that, as he now understands things, would have tended to destroy the world rather than to save it. Due to insufficient rationality, apparently.

I’ve practiced a fair amount of (2), but much less than I could imagine some practicing; and, as I noted in the comment Yvain cited, it seems to have done me some good. Broadly similar results for the handful of others I know who try to get rationality into their bones. Less impressive than I’d like, but I tend to interpret this a a sign we should spend more time on skateboards, and I anticipate that we’ll see more real improvement as we do.

The most important actual helps involve that topic we’re not supposed to discuss here until May, but I’d say we were able to choose a much higher-impact way to help the world than people without x-rationality standardly choose, and that we’re able to actually think usefully about a subject where most conversations degenerate into storytelling, availability heuristics, attaching overmuch weight to specific conjunctions, etc. Which, if there’s any non-negligible chance we’re right, is immensely practical. But we’re also somewhat better at strategicness about actually exercising, about using social interaction patterns that work better than the ones we were accidentally using previously (though far from as well as the ones the best people use), about choosing college or career tracks that have better expected results, etc.

Folks with more data here (positive or negative), please share.

Regarding claim (1):
I guess I wouldn’t be surprised by anything from “massive practical help, at least from particular skilled/lucky dojos that get on good tracks” to “not much help at all”. But if we do get “not much help at all”, I’ll feel like there was a thing we could have done, and we didn’t manage to do it. There are loads of ridiculously stupid kinds of decision-making that most people do, and it would be strange if there were no way we could get visible practical benefit from improving on that. Details in later comments.

Comment author: Yvain 09 April 2009 12:27:38PM *  5 points [-]

I agree with almost everything here, with the following caveats:

I. The practical benefits we get from (3) are (I think I'm agreeing with you here) likely to be so small as to be difficult to measure informally; i.e. anyone who claims to have noticed a specific improvement is as likely to be imagining it as really improving. Probably some effects that could be measured in a formal experiment with a very large sample size, but this is not what we have been doing.

II. (2) shows promise but is not something I see discussed very often on Overcoming Bias or Less Wrong. Using the Boyle metaphor, this would be the technology of rationality, as opposed to the science of it. I've seen a few suggestions for "techniques", but they seem sort of ad hoc (I will admit, in retrospect, that many of the times I was proposing 'techniques' were more of an attempt to sound like I was thinking pragmatically, than soundly based on good experimental evidence). I've tried to apply specific methods to specific decisions, but never gone so far as to set aside a half hour each day to "rationality practice", nor would I really know what to do with that half hour if I did. I'd like to know more about what you do and what you think has helped.

III. You list a greater appreciation of transhumanism as one of the benefits of x-rationality, but the causal linkage doesn't impress me. Many of the transhumanists here were transhumanists before they were rationalists, and only came to Overcoming Bias out of interest in reading what transhumanist leaders Eliezer and Robin had to say. I think my "conversion" to transhumanism came about mostly because I started meeting so many extremely intelligent transhumanists that it no longer seemed like a fringe crazy-person belief and my mind felt free to judge it with the algorithms it uses for normal scientific theories rather than the algorithms it uses for random Internet crackpottery. Many other OB readers came to transhumanism just because EY and RH explicitly argued for it and did a good job. Still others probably felt pressure to "convert" as an in-group identification thing. And finally, I think transhumanists and x-rationalists are part of that big atheist/libertarian/sci-fi/et cetera personspace cluster Eliezer's been talking about: we all had a natural vulnerability to that meme before ever arriving here. AFAIK Kahneman and Tversky are not transhumanists, Aumann certainly isn't, and I would be surprised if x-rationalists not associated with EY and RH and our group come to transhumanism in numbers greater than their personspace cluster membership predicts.

IV. Given fifty years to improve the Art, I also wouldn't be surprised with anything from "massive practical help" to "not much help at all". I don't know exactly what you mean by "ridiculously stupid decision-making that most people do", but are you sure it's something that should be solved with x-rationality as opposed to normal rationality?

Comment author: simpleton 09 April 2009 04:22:51AM *  17 points [-]

If in 1660 you'd asked the first members of the Royal Society to list the ways in which natural philosophy had tangibly improved their lives, you probably wouldn't have gotten a very impressive list.

Looking over history, you would not have found any tendency for successful people to have made a formal study of natural philosophy.

Comment author: Yvain 09 April 2009 04:48:08AM *  21 points [-]

It would be overconfident for me to say rationality could never become useful. My point is just that we are acting like it's practically useful right now, without very much evidence for this beyond our hopes and dreams. Thus my last sentence - that "crossing the Pacific" isn't impossible, but it's going to take a different level of effort.

If in 1660, Robert Boyle had gone around saying that, now that we knew Boyle's Law of gas behavior, we should be able to predict the weather, and that that was the only point of discovering Boyle's Law and that furthermore we should never trust a so-called chemist or physicist except insofar as he successfully predicted the weather - then I think the Royal Society would be making the same mistake we are.

Boyle's Law is sort of helpful in understanding the weather, sort of. But it's step one of ten million steps, used alone it doesn't work nearly as well as just eyeballing the weather and looking for patterns, and any attempt to judge applicants to the Royal Society on their weather prediction abilities would have excluded some excellent scientists. Any attempt to restrict gas physics itself to things that were directly helpful in predicting the weather would have destroyed the science, ironically including the discoveries two hundred years down the road that were helpful in weather prediction.

Summed up: With luck, (some) science can result in good practical technology. But demanding the technology too soon, or restricting science to only the science with technology to back it up, hurts both science and technology.

(there is a difference between verification and technology. Boyle was able to empirically test his gas law, but not practically apply it. This may be fuzzier in rationality)

Comment author: jimrandomh 09 April 2009 03:27:38AM *  4 points [-]

Extreme rationality is for important decisions, not for choosing your breakfast cereal. Really important decisions - by which I mean those that you'd sleep on, and allocate more than ten minutes of thought - typically coincide with changes in habits and routine, which don't happen more often than once in several months. For more common decisions, we only have time and energy for ordinary rationality.

Comment author: Yvain 09 April 2009 03:46:59AM 4 points [-]

I agree with this, but I also think that our big important decisions probably determine a lot less of our success than we like to think. A very large part of success probably comes from either the sum of our smaller decisions, or from decisions that didn't seem too important at the time but ended up making a very large difference in retrospect. The experiment I mentioned has raised my awareness of this.

I also think the big decisions are the ones it's hardest to apply extreme rationality to, both because the emotional stakes are so high and because by the time we make them we've already made a pile of smaller decisions that have tipped us in one or the other direction. See http://www.overcomingbias.com/2007/10/we-change-our-m.html . I predict not-significantly-different statistics for people who have trained in extreme rationality, though without a very high degree of confidence.

Extreme Rationality: It's Not That Great

140 Post author: Yvain 09 April 2009 02:44AM

Related to: Individual Rationality is a Matter of Life and Death, The Benefits of Rationality, Rationality is Systematized Winning
But I finally snapped after reading: Mandatory Secret Identities

Okay, the title was for shock value. Rationality is pretty great. Just not quite as great as everyone here seems to think it is.

For this post, I will be using "extreme rationality" or "x-rationality" in the sense of "techniques and theories from Overcoming Bias, Less Wrong, or similar deliberate formal rationality study programs, above and beyond the standard level of rationality possessed by an intelligent science-literate person without formal rationalist training." It seems pretty uncontroversial that there are massive benefits from going from a completely irrational moron to the average intelligent person's level. I'm coining this new term so there's no temptation to confuse x-rationality with normal, lower-level rationality.

And for this post, I use "benefits" or "practical benefits" to mean anything not relating to philosophy, truth, winning debates, or a sense of personal satisfaction from understanding things better. Money, status, popularity, and scientific discovery all count.

So, what are these "benefits" of "x-rationality"?

A while back, Vladimir Nesov asked exactly that, and made a thread for people to list all of the positive effects x-rationality had on their lives. Only a handful responded, and most responses weren't very practical. Anna Salamon, one of the few people to give a really impressive list of benefits, wrote:

I'm surprised there are so few apparent gains listed. Are most people who benefited just being silent? We should expect a certain number of headache-cures, etc., just by placebo effects or coincidences of timing.

There have since been a few more people claiming practical benefits from x-rationality, but we should generally expect more people to claim benefits than to actually experience them. Anna mentions the placebo effect, and to that I would add cognitive dissonance - people spent all this time learning x-rationality, so it MUST have helped them! - and the same sort of confirmation bias that makes Christians swear that their prayers really work.

I find my personal experience in accord with the evidence from Vladimir's thread. I've gotten countless clarity-of-mind benefits from Overcoming Bias' x-rationality, but practical benefits? Aside from some peripheral disciplines1, I can't think of any.

Looking over history, I do not find any tendency for successful people to have made a formal study of x-rationality. This isn't entirely fair, because the discipline has expanded vastly over the past fifty years, but the basics - syllogisms, fallacies, and the like - have been around much longer. The few groups who made a concerted effort to study x-rationality didn't shoot off an unusual number of geniuses - the Korzybskians are a good example. In fact as far as I know the only follower of Korzybski to turn his ideas into a vast personal empire of fame and fortune was (ironically!) L. Ron Hubbard, who took the basic concept of techniques to purge confusions from the mind, replaced the substance with a bunch of attractive flim-flam, and founded Scientology. And like Hubbard's superstar followers, many of this century's most successful people have been notably irrational.

There seems to me to be approximately zero empirical evidence that x-rationality has a large effect on your practical success, and some anecdotal empirical evidence against it. The evidence in favor of the proposition right now seems to be its sheer obviousness. Rationality is the study of knowing the truth and making good decisions. How the heck could knowing more than everyone else and making better decisions than them not make you more successful?!?

This is a difficult question, but I think it has an answer. A complex, multifactorial answer, but an answer.

One factor we have to once again come back to is akrasia2. I find akrasia in myself and others to be the most important limiting factor to our success. Think of that phrase "limiting factor" formally, the way you'd think of the limiting reagent in chemistry. When there's a limiting reagent, it doesn't matter how much more of the other reagents you add, the reaction's not going to make any more product. Rational decisions are practically useless without the willpower to carry them out. If our limiting reagent is willpower and not rationality, throwing truckloads of rationality into our brains isn't going to increase success very much.

This is a very large part of the story, but not the whole story. If I was rational enough to pick only stocks that would go up, I'd become successful regardless of how little willpower I had, as long as it was enough to pick up the phone and call my broker.

So the second factor is that most people are rational enough for their own purposes. Oh, they go on wild flights of fancy when discussing politics or religion or philosophy, but when it comes to business they suddenly become cold and calculating. This relates to Robin Hanson on Near and Far modes of thinking. Near Mode thinking is actually pretty good at a lot of things, and Near Mode thinking is the thinking whose accuracy gives us practical benefits.

And - when I was young, I used to watch The Journey of Allen Strange on Nickleodeon. It was a children's show about this alien who came to Earth and lived with these kids. I remember one scene where Allen the Alien was watching the kids play pool. "That's amazing," Allen told them. "I could never calculate differential equations in my head that quickly." The kids had to convince him that "it's in the arm, not the head" - that even though the movement of the balls is governed by differential equations, humans don't actually calculate the equations each time they play. They just move their arm in a way that feels right. If Allen had been smarter, he could have explained that the kids were doing some very impressive mathematics on a subconscious level that produced their arm's perception of "feeling right". But the kids' point still stands; even though in theory explicit mathematics will produce better results than eyeballing it, in practice you can't become a good pool player just by studying calculus.

A lot of human rationality follows the same pattern. Isaac Newton is frequently named as a guy who knew no formal theories of science or rationality, who was hopelessly irrational in his philosophical beliefs and his personal life, but who is still widely and justifiably considered the greatest scientist who ever lived. Would Newton have gone even further if he'd known Bayes theory? Probably it would've been like telling the world pool champion to try using more calculus in his shots: not a pretty sight.

Yes, yes, beisutsukai should be able to develop quantum gravity in a month and so on. But until someone on Less Wrong actually goes and does it, that story sounds a lot like when Alfred Korzybski claimed that World War Two could have been prevented if everyone had just used more General Semantics.

And then there's just plain noise. Your success in the world depends on things ranging from your hairstyle to your height to your social skills to your IQ score to cognitive constructs psychologists don't even have names for yet. X-Rationality can help you succeed. But so can excellent fashion sense. It's not clear in real-world terms that x-rationality has more of an effect than fashion. And don't dismiss that with "A good x-rationalist will know if fashion is important, and study fashion." A good normal rationalist could do that too; it's not a specific advantage of x-rationalism, just of having a general rational outlook. And having a general rational outlook, as I mentioned before, is limited in its effectiveness by poor application and akrasia.

I no longer believe mastering all these Overcoming Bias and Less Wrong techniques will turn me into Anasûrimbor Kellhus or John Galt. I no longer even believe mastering all these Overcoming Bias techniques will turn me into Eliezer Yudkowsky (who, as his writings from 2001 indicate, had developed his characteristic level of awesomeness before he became interested in x-rationality at all)3. I think it may help me succeed in life a little, but I think the correlation between x-rationality and success is probably closer to 0.1 than to 1. Maybe 0.2 in some businesses like finance, but people in finance tend to know this and use specially developed x-rationalist techniques on the job already without making it a lifestyle commitment. I think it was primarily a Happy Death Spiral around how wonderfully super-awesome x-rationality was that made me once think otherwise.

And this is why I am not so impressed by Eliezer's claim that an x-rationality instructor should be successful in their non-rationality life. Yes, there probably are some x-rationalists who will also be successful people. But again, correlation 0.1. Stop saying only practically successful people could be good x-rationality teachers! Stop saying we need to start having huge real-life victories or our art is useless! Stop calling x-rationality the Art of Winning! Stop saying I must be engaged in some sort of weird signalling effort for saying I'm here because I like mental clarity instead of because I want to be the next Bill Gates! It trivializes the very virtues that brought most of us to Overcoming Bias, and replaces them with what sounds a lot like a pitch for some weird self-help cult...

...

...

...but you will disagree with me. And we are both aspiring rationalists, and therefore we resolve disagreements by experiments. I propose one.

For the next time period - a week, a month, whatever - take special note of every decision you make. By "decision", I don't mean the decision to get up in the morning, I mean the sort that's made on a conscious level and requires at least a few seconds' serious thought. Make a tick mark, literal or mental, so you can count how many of these there are.

Then note whether you make that decision rationally. If yes, also record whether you made that decision x-rationally. I don't just mean you spent a brief second thinking about whether any biases might have affected your choice. I mean one where you think there's a serious (let's arbitrarily say 33%) chance that using x-rationality instead of normal rationality actually changed the result of your decision.

Finally, note whether, once you came to the rational conclusion, you actually followed it. This is not a trivial matter. For example, before writing this blog post I wondered briefly whether I should use the time studying instead, used normal (but not x-) rationality to determine that yes, I should, and then proceeded to write this anyway. And if you get that far, note whether your x-rational decisions tend to turn out particularly well.

This experiment seems easy to rig4; merely doing it should increase your level of conscious rational decisions quite a bit. And yet I have been trying it for the past few days, and the results have not been pretty. Not pretty at all. Not only do I make fewer conscious decisions than I thought, but the ones I do make I rarely apply even the slightest modicum of rationality to, and the ones I apply rationality to it's practically never x-rationality, and when I do apply everything I've got I don't seem to follow those decisions too consistently.

I'm not so great a rationalist anyway, and I may be especially bad at this. So I'm interested in hearing how different your results are. Just don't rig it. If you find yourself using x-rationality twenty times more often than you were when you weren't performing the experiment, you're rigging it, consciously or otherwise5.

Eliezer writes:

The novice goes astray and says, "The Art failed me."
The master goes astray and says, "I failed my Art."

Yet one way to fail your Art is to expect more of it than it can deliver. No matter how good a swimmer you are, you will not be able to cross the Pacific. This is not to say crossing the Pacific is impossible. It just means it will require a different sort of thinking than the one you've been using thus far. Perhaps there are developments of the Art of Rationality or its associated Arts that can turn us into a Kellhus or a Galt, but they will not be reached by trying to overcome biases really really hard.

Footnotes:

1: Specifically, reading Overcoming Bias convinced me to study evolutionary psychology in some depth, which has been useful in social situations. As far as I know. I'd probably be biased into thinking it had been even if it hadn't, because I like evo psych and it's very hard to measure.

2: Eliezer considers fighting akrasia to be part of the art of rationality; he compares it to "kicking" to our "punching". I'm not sure why he considers them to be the same Art rather than two related Arts.

3: This is actually an important point. I think there are probably quite a few smart, successful people who develop an interest in x-rationality, but I can't think of any people who started out merely above-average, developed an interest in x-rationality, and then became smart and successful because of that x-rationality.

4: This is a terribly controlled experiment, and the only way its data can be meaningfully interpreted at all is through what one of my professors called the "ocular trauma test" - when the data hits you between the eyes. If people claim they always follow their rational decisions, I think I will be more likely to interpret it as lack of enough cognitive self-consciousness to notice when they're doing something irrational than an honest lack of irrationality.

5: In which case it will have ceased to be an experiment and become a technique instead. I've noticed this happening a lot over the past few days, and I may continue doing it.

Comment author: Eliezer_Yudkowsky 08 April 2009 09:17:14PM 3 points [-]

I don't think I can afford to divert my attention into setting one up, but I've heard others already discussing it, so it's worth placing some Go stones around it.

Comment author: Yvain 08 April 2009 09:19:24PM 2 points [-]

Really? If it's not too private, who's been discussing it?

Comment author: Eliezer_Yudkowsky 08 April 2009 09:07:11PM 4 points [-]

(Blinks.)

I wonder if this idea comes as a shock because everyone was planning on becoming rationality instructors, i.e., I should have warned everyone about this much earlier?

Is it offputting on some other level?

But I must also consider that it might really be that stupid. Damn, now I wish I knew the actual number of upvotes and downvotes!

Comment author: Yvain 08 April 2009 09:13:52PM 5 points [-]

I don't think too many people are actually considering "rationality instructor" as a career path at this point - which reminds me - what exactly are your plans for this rationality dojo thing anyway? Is it just something you like to talk about, or something you plan to one day set up? Are you hoping people from Less Wrong will start the first ones, or that people from Less Wrong will be students in ones set up in some other way?

Comment author: orthonormal 08 April 2009 05:20:18PM 5 points [-]

I did think about Marxism, but why would it be less likely to explode in my face? It's also a modern political position.

The probability of a staunch Marxist commenting on this blog seems to be vastly less than the probability of a staunch (non-religious) Republican doing so. We don't want to drive away many potential readers on account of their surface positions before they have a chance to reconsider things. Also, we really don't want an accidental political flamewar to start in a comment thread, so political examples should be chosen to tread on fewer current toes if at all possible.

Would you prefer that next time I include two examples, one attacking either "side" of the political "spectrum" next time?

No! This would make it even more likely to cause one of the bad outcomes above! This isn't about "fairness", it's about prudence.

Or can you think of some historical example that would be as immediately recognizable to everyone here as the Republican one?

Again, I think people here generally have the basic political literacy to recognize the Marxist example.

Comment author: Yvain 08 April 2009 09:05:59PM 9 points [-]

I spend most of my time abroad, and come across more (or at least louder) Marxist sympathizers than conservative Republican sympathizers. This is probably not representative of people on this blog, and I will take it into account next time I post something.

Comment author: Eliezer_Yudkowsky 08 April 2009 08:03:33PM 14 points [-]

If you were to say tomorrow "I've been lying about the whole AI programmer thing; I actually live in my parents' basement and have never done anything worthwhile in any non-rationality field in my entire life," then would I have to revise my opinion that you're a very good rationality teacher? Would I have to deny having learned really valuable things from you?

But the fact that reality doesn't disentangle this way, is in a sense the whole point - it's not a coincidence that things are the way they are.

If we get far enough to have external real-world standards like those you're describing, then yes we can toss the "secret identity" thing out the window, so long as we don't have the problem of most good students wanting only to become rationality instructors themselves as opposed to going into other careers (but a teacher who raised their students this way would suffer on the 'accomplished students' metric, etc.). But on the other hand I still suspect that the instructors with secret identities would be revealed to do better.

Comment author: Yvain 08 April 2009 08:54:36PM 4 points [-]

Right. Our difference of opinion here is clearly nontrivial. I'll put it on the list of things to write posts about.

Comment author: Yvain 08 April 2009 07:46:13PM *  31 points [-]

What does this post even mean? I don't have access to my own respect function, and I don't know if I'd mess with it this way even if I did.

If you were to say tomorrow "I've been lying about the whole AI programmer thing; I actually live in my parents' basement and have never done anything worthwhile in any non-rationality field in my entire life," then would I have to revise my opinion that you're a very good rationality teacher? Would I have to deny having learned really valuable things from you?

Or would I have to say, "Well, this guy named Eliezer taught me everything I know, he's completely opened my mind to new domains of knowledge, and you should totally read everything he's written - but he's not all that great and I don't have any respect for him and you shouldn't either" when referring people to your writing?

Or to put it another way...let's say there are two rationality instructors in my city. One, John, is a world famous physicist, businessman, and writer. The other, Mary, has no particular accomplishments outside her rationality instruction work. However, Mary's students have been observed to do much better at their careers than John's, and every time the two dojos go up against each other in Rationalist Debating or calibration tests or any other kind of measurement, Mary's students do better. Wouldn't it be, well, irrational for me to go to John's dojo instead of Mary's? Would the Bayesian Police have to surround Mary's dojo and make sure her students don't say nice things about her or pay her more money than John is making?

Comment author: Yvain 08 April 2009 04:54:35PM 6 points [-]

I'd write a list of my favorite books and then erase my memories of them, so I could read them again for the first time.

Comment author: PhilGoetz 08 April 2009 04:29:02AM *  5 points [-]

Likewise, when Christians talk about persecution, they usually point out that one great way to stop this persecution would be to put up the Ten Commandments in all public places.

Citation, please? I've heard a fair amount of such talk, and don't ever recall the 10 Commandments being proposed as a solution.

Comment author: Yvain 08 April 2009 03:00:37PM 2 points [-]

A quick Google search for "Christian persecution ten commandments" gives articles like this one where the author gives the "trashing of the ten commandments" and the inability to put them up in public places as evidence of the persecution of Christians. The point seems to be that if we want to reverse this persecution of Christians, we need to stop preventing them from putting the 10 Commandments up.

I'll admit this is less explicit than "If you want to stop persecuting us, you must put up 10 commandments everywhere".

Comment author: orthonormal 08 April 2009 02:19:29AM 11 points [-]

From Politics is the Mind-Killer:

If your point is inherently about politics, then talk about Louis XVI during the French Revolution.

Or how about Marxism at least? Exact same dynamic as the one you speak of (the claim that they speak for a silent oppressed majority that's certain to win out in the end), but far less likely to explode in your face on this blog.

Comment author: Yvain 08 April 2009 02:55:36PM *  9 points [-]

I did think about Marxism, but why would it be less likely to explode in my face? It's also a modern political position. I decided to go with the Republican case because it was where I originally noticed it and as far as I can tell the most archetypal example. I considered it acceptable because I'm not actually saying the Republicans are wrong about any particular policy issue.

Would you prefer that next time I include two examples, one attacking either "side" of the political "spectrum" next time? Or can you think of some historical example that would be as immediately recognizable to everyone here as the Republican one?

I'm also getting a little sick of always using groups disliked by the entire Less Wrong community as examples (eg Christians). Yes, it makes it easier to read without getting angry, but it seems too potentially dangerous to come here and see something else accusing Christians every night. I don't know what to do about it.

Comment author: steven0461 08 April 2009 01:52:46PM 0 points [-]

Huh? Surely, by the definition of "oppress", A can oppress B only if A is in the wrong. Who would ever say things like "the Israelis are oppressing the Palestinians, and they are right to do so"?

Comment author: Yvain 08 April 2009 02:46:54PM 4 points [-]

The connotation is that it's bad, yes. Reduce it to "The Israelis are seriously restricting the opportunities of the Palestinians" and probably there are many Israel supporters who would agree it's true but say Israel's not in the wrong. The use of the word "oppress" is how you would express "seriously restricting opportunities" if you wanted to communicate that it was bad.

Comment author: taw 08 April 2009 12:23:47AM 0 points [-]

I'd say it's mostly #2 - people do have strong sense of fairness. Unfairness happens a lot, but it's hard to measure objectively.

So groups talking a lot about injustices happening to their members, even exaggerating them, are better off that groups taking injustices silently, so there's really no incentive to keep quiet.

It's not really something that can be proven or disproven by data convincingly most of the time, even in pretty extreme cases that could be measured (like Palestinians, which are genuinely oppressed by any objective measure) you can quite successfully rationalize the injustice away with good publicity machine ("that's because they're terrorists" or something).

Comment author: Yvain 08 April 2009 12:29:29AM 4 points [-]

It's not really something that can be proven or disproven by data convincingly most of the time, even in pretty extreme cases that could be measured (like Palestinians, which are genuinely oppressed by any objective measure) you can quite successfully rationalize the injustice away with good publicity machine ("that's because they're terrorists" or something).

I think a lot of it has to do with just how closely linked powerlessness and rightness are in people's minds. If someone supporting Palestine tells someone supporting Israel that the Palestinians are oppressed, the Israel supporter feels like letting this statement go would be equivalent to admitting that Palestinians are in the right and Israel is in the wrong, or even that Israel doesn't have a right to exist.

The best way to deal with this IMHO would be to say "Yes, the Palestinians are oppressed, but even so Israel still deserves its land." But this is a very difficult statement to make. The natural human response is "No, the Palestinians aren't oppressed!"

Help, help, I'm being oppressed!

30 Post author: Yvain 07 April 2009 11:22PM

Followup toWhy Support the Underdog?
Serendipitously related to: Whining-Based Communities

Pity whatever U.N. official has to keep track of all the persecution going on. With two hundred plus countries in the world, there's just so much of it.

Some places persecute Christians. Here's a Christian writer from a nation we'll call Country A:

Global reports indicate that over 150,000 Christians were martyred last year, chiefly in foreign countries. However, statistics are changing: persecution of Christians is on the increase at home. What's happening to bring about this change? According to some experts a pattern is emerging reminiscent of Jewish persecution in post war Germany. "Isolation of, and discrimination against Christians is growing almost geometrically" says Don McAlvany in The Midnight Herald. "This is the way it started in Germany against the Jews. As they became more isolated and marginalized by the Nazi propaganda machine, as popular hatred and prejudice against the Jews increased among the German people, wholesale persecution followed.  Could this be where the growing anti-Christian consensus in this country is taking us?"

And some countries persecute atheists. Here's an atheist activist describing what we'll call Country B.

Godless atheists are the most despised and distrusted minority in our country. The growing attention to atheism and atheists has given rise to increased anti-atheist bigotry in the media. Circumstances for them can be difficult enough that they have to stay in the closet and hide their atheism from friends and family. Atheists have to fear discrimination on the job, in the community, and even in their own families if their atheism is made known. Some even have to contend with harassment and vandalism. Distrust and hatred of atheists is widespread enough through our society that they have plenty of reasons to be concerned.

Some countries persecute Muslims. A Muslim youth in Country C:

The government has continuously persecuted Arabs and Muslims with extremist and unpopular views, charging them with terrorism and criminal acts related to terrorism. I am proud of [Muslims] who stand up to this system of injustice and to our country's gulag. They may beat them, but they will continues to suffer because in this country, Arabs are never innocent, they are merely guilty of lesser crimes. Even if they are proven innocent, after years of suffering and being defamed, the gulag and the political persecution will continue.

And some countries persecute everyone except Muslims. A politician in Country D writes:

The gathering storm I have been warning of for years has now formed over us. Yet instead of fighting the gradual incursion of Sharia and the demands of an intolerant, even militant Islam, we are cowering and fatalistic.

Since countries A, B, C, and D are all America1, what's up with all these people claiming persecution?

I don't doubt that there are examples of Christians, atheists, Muslims, and non-Muslims all getting persecuted in the US. There's no rule that says only one group can be persecuted at a time, especially in a society as pluralistic as our own. But compare the claim "There are a few incidents of people persecuting Christians" with the claim "Christians are a persecuted group in our society." The first reduces to an objectively true statement. The second is a sorta-meaningless "dangling variable" that can be declared either true or false depending on what connotation you want to send.

And people tend to take the liberty to call the is_persecuted variable "true" for their own group and "false" for groups they don't like. Why does everyone want to be persecuted so badly? Here are some reasons I can think of:

1. The tendency to support the underdog. Being persecuted is about as underdog as you can get, and underdog supporters everywhere are quick to leap to the support of persecuted groups.

2. To create an incentive for fair-minded people to "level the playing field" by raising their status. I read about a tribe in India involved in a media campaign to inform everyone just how persecuted they really were. Why? They wanted to be added to India's affirmative action program, which would give them a better chance at government jobs. Likewise, when Christians talk about persecution, they usually point out that one great way to stop this persecution would be to put up the Ten Commandments in all public places.

3. To self-handicap. If I'm unsuccessful, it's not because I'm lazy or unqualified, it's beacuse they were persecuting me! Likewise, if I'm successful, then I managed to triumph in the face of adversity. I'm practically Martin Luther King or someone.

4. To build in-group cohesiveness. People come together in the face of a common enemy.

5. To explain away a lack of success. Let's say you're a fundamentalist Christian and you notice most of the rest of America dislikes you and thinks you're crazy. You might say "Well, by Aumann's Agreement Theorem, they probably know something I don't, and I should moderate my religious views." But if your Revolutionary is AWOL, your Apologist could conclude that there is a sinister campaign going on to discredit Christianity, and everyone has fallen for this campaign but you and your friends.

I think these all play a role, with 1 and 2 the most important.

But one common thread in psychology is that the mind very frequently wants to have its cake and eat it too. Last week, we agreed that people like supporting the underdog, but we also agreed that there's a benefit to being on top; that when push comes to shove a lot of people are going to side with Zug instead of Urk. What would be really useful in winning converts would be to be a persecuted underdog who was also very powerful and certain to win out. But how would you do that?

Some Republicans have found a way. Whether they're in control of the government or not, the right-wing blogosphere invariably presents them as under siege, a rapidly dwindling holdout of Real American Values in a country utterly in the grip of liberalism.

But they don't say anything like "Everyone's liberal, things are hopeless, might as well stay home." They believe in a silent majority. Liberals control all sorts of nefarious institutions that are currently exercising a stranglehold on power and hiding the truth, but most Americans, once you pull the wool off their eyes, are conservatives at heart and just as angry about this whole thing as they are. Any day now, they're going to throw off the yoke of liberal tyranny and take back their own country.

This is a great system. Think about it. Not only should you support the Republicans for support-the-underdog and level-the-playing-field reasons, you should also support them for majoritarian reasons and because their side has the best chance of winning. It's the best possible world short of coming out and saying "Insofar as it makes you want to vote for us, we are in total control of the country, but insofar as that makes you not want to vote for us, we are a tiny persecuted minority who need your help".

We're coming dangerously close to talking politics here, but this isn't just a Republican phenomenon. It underlies a lot of the uses of the word "elite" - this sense that there's a small minority of wrong-headed people who disagree with you in control of everything, even though the vast majority of people are secretly on your side. Whether it's the "neoliberal capitalist elite", the "east coast intellectual elite" or whatever, it's a one word Pavlovian trigger that activates this concept of your favorite group simultaneously being dominant and being persecuted by those darned elites.

There are branches of social science that consciously devote themselves solely to officially identifying the Powerful and the Powerless in every issue and conflict. They have their uses. But as rationalists, we need to devote ourselves to the separate task of disentangling the question at hand from the question of who is more powerful. Otherwise, we are at the mercy of the underdog bias, the support-the-winning-team bias, and any mutant combinations of them that may arise2.

As is often the case, reduction of statements with objective truth-values can save your hide here. If every time Chris the Christian says "Christians are persecuted," you hear "Christians aren't allowed to stick the Ten Commandments up in schools," then you're no longer vulnerable to his appeal to pity.

What other defenses are there against the human tendency to obsess over which side is more powerful, instead of which side is right?

Footnotes:

1: The first comment comes from Worthy News, the second from About Atheism, the third from Mideast Youth, and the fourth is Senator Rick Santorum

2: Has anyone else ever watched two people in an argument completely abandon discussion over who is right, and instead turn to which person's side is persecuted worse, as if they were more or less the same question anyway? It's not a pretty sight.

Comment author: HughRistik 07 April 2009 05:15:31AM *  2 points [-]

Shall I take the lack of comments to mean that:

a) Everyone agrees with my points and thinks that they are obvious or trivial

b) Nobody is interested in this topic

c) TL:DR

d) This post is very general and people are waiting for me to get more specific

e) Nobody knows what heuristic is, so they don't read the post

f) ???

Comment author: Yvain 07 April 2009 05:33:15AM *  4 points [-]

Mostly D. A little A, but mostly the first, less negative half of A.

Comment author: Yvain 06 April 2009 06:27:32PM 6 points [-]

IAWYC, but the reason I was interested in evolutionary explanations was twofold: first, that a lot of standard (nonevolutionary) psychologists had attacked the problem and come up with what I considered unsatisfying explanations; and second, that this seemed like exactly the sort of area evolutionary psychology had been successful at explaining in the past (ie, a universal human tendency relating to strategies in conflicts and potentially having a large impact on future success).

I don't know what to think about the proposed solutions, including Roko's. On the one hand, they all sound pretty good, including the non-evolutionary ones. On the other hand, they all sound pretty good. Although it's always possible that there was more than one pressure driving people to support the underdog, five or six separate ones working simultaneously is a bit of a stretch. That means I probably have a low standard for "sounds pretty good". Which might be your point.

Still, I don't know what you want us to do. Are you just saying keep a low probability for all untested hypotheses? That sounds like a pretty good idea.

Comment author: AlexU 06 April 2009 01:42:53AM 5 points [-]

Shouldn't the fact that they can probably imagine better versions of themselves reduce this probability? If you're in a holodeck, in addition to putting yourself at the center of the Singularity, why wouldn't you also give yourself the looks of Brad Pitt and the wealth of Bill Gates?

Comment author: Yvain 06 April 2009 09:50:49AM *  5 points [-]
  1. No if they are in a historical simulation. The real architects of the Singularity weren't billionaires.
  2. No if they are in some kind of holo-game, for the same reason that people playing computer games don't hack them to make their character level infinity and impervious to bullets. Where would be the fun in that?
Comment author: James_Miller 05 April 2009 11:01:28PM 14 points [-]

So with what probability should Barack Obama believe he is on a holodeck, and how should this belief influence his behavior?

Comment author: Yvain 05 April 2009 11:43:53PM *  17 points [-]

And not only Obama. The closer you are to the center of human history, the more likely you are to be on a holodeck. People simulating others should be more likely to simulate people in historically interesting times, and people simulating themselves for fun and blocking their memory should be more likely to simulate themselves as close to interesting events as possible.

And...if Singularity theory is true, the Singularity will be the most interesting and important event in all human history. Now, all of us are suspiciously close to the Singularity, with a suspiciously large ability to influence its course. Even I, a not-too-involved person who's just donated a few hundred dollars to SIAI and gets to sit here talking to the SIAI leadership each night, am probably within the top millionth of humans who have ever lived in terms of Singularity "proximity".

And Michael Vassar and Eliezer are so close to the theorized center of human history that they should assume they're holodecking with probability ~1.

After all, which is more likely from their perspective - that they're one of the dozen or so people most responsible for creating the Singularity and ensuring Friendly AI, or that they're some posthuman history buff who wanted to know what being the guy who led the Singularity Institute was like?

(the alternate explanation, of course, is that we're all on the completely wrong track and that we're simply in the larger percentage of humans who think they're extremely important.)

Comment author: gwern 05 April 2009 01:00:13AM *  4 points [-]

The proffered explanations seem plausible. What about with ideas though? I think it's social signaling: 'Look how clever and independent and different I am, that I can adopt this minority viewpoint and justify it.'

(Kind of like Zahavi's handicap principle.)

EDIT: It appears I largely stole this variant on signaling strategy from http://www.overcomingbias.com/2008/12/showoff-bias.html . Oh well.

Comment author: Yvain 05 April 2009 01:20:46AM 10 points [-]

Your mention of signaling gives me an idea.

What if the mechanism isn't designed to actually support the underdog, but to signal a tendency to support the underdog?

In a world where everyone supports the likely winner, Zug doesn't need to promise anyone anything to keep them on his side. But if one person suddenly develops a tendency to support the underdog, then Zug has to keep him loyal by promising him extra rewards.

The best possible case is one where you end up on Zug's side, but only after vacillating for so long that Zug is terrified you're going to side with Urk and promises everything in his power to win you over. And the only way to terrify Zug that way is to actually side with Urk sometimes.

Why Support the Underdog?

35 Post author: Yvain 05 April 2009 12:01AM

One of the strangest human biases is the almost universal tendency to support the underdog.

I say "human" because even though Americans like to identify themselves as particular friends of the underdog, you can find a little of it everywhere. Anyone who's watched anime knows the Japanese have it. Anyone who's read the Bible knows the Israelites had it (no one was rooting for Goliath!) From mythology to literature to politics to sports, it keeps coming up.

I say "universal" because it doesn't just affect silly things like sports teams. Some psychologists did a study where they showed participants two maps of Israel: one showing it as a large country surrounding the small Palestinian enclaves, and the other showing it as a tiny island in the middle of the hostile Arab world. In the "Palestinians as underdogs" condition, 55% said they supported Palestine. In the "Israelis as underdogs" condition, 75% said they supported Israel. Yes, you can change opinion thirty points by altering perceived underdog status. By comparison, my informal experiments trying to teach people relevant facts about the region's history changed opinion approximately zero percent.

(Oh, and the Israelis and Palestinians know this. That's why the propaganda handbooks they give to their respective supporters - of course they give their supporters propaganda handbooks! - specifically suggest the supporters portray their chosen cause as an underdog. It's also why every time BBC or someone shows a clip about the region, they get complaints from people who thought it didn't make their chosen side seem weak enough!)

And there aren't many mitigating factors. Even when the underdog is obviously completely doomed, we still identify with them: witness Leonidas at Thermopylae. Even when the underdog is evil and the powerful faction is good, we can still feel a little sympathy for them; I remember some of my friends and I talking about bin Laden, and admitting that although he was clearly an evil terrorist scumbag, there was still something sort of awesome about a guy who could take on the entire western world from a cave somewhere.

I say "strangest" because I can't make heads or tails of why evolutionary psychology would allow it. Let's say Zug and Urk are battling it out for supremacy of your hunter-gatherer tribe. Urk comes to you and says "Hey, my faction is really weak. We don't have a chance against Zug, who is much stronger than us. I think we will probably be defeated and humiliated, and our property divided up among Zug's supporters."

The purely rational response seems to be "Wow, thanks for warning me, I'll go join Zug's side right now. Riches and high status as part of the winning faction, here I come!"

Now, many of us probably would join Zug's side. But introspection would tell us we were opposing rational calculation on Zug's side to a native, preconscious support for Urk. Why? The native preconscious part of our brain is usually the one that's really good at ending up on top in tribal power struggles. This sort of thing goes against everything it usually stands for.

I can think of a few explanations, none of them satisfying. First, it could be a mechanism to prevent any one person from getting too powerful. Problem is, this sounds kind of like group selection. Maybe the group does best if there's no one dictator, but from an individual point of view, the best thing to do in a group with a powerful dictator is get on that dictator's good side. Any single individual who initiates the strategy of supporting the underdog gets crushed by all the other people who are still on the dictator's team.

Second, it could be a mechanism to go where the rewards are highest. If a hundred people support Zug, and only ten people support Urk, then you have a chance to become one of Urk's top lieutenants, with all the high status and reproductive opportunities that implies if Urk wins. But I don't like this explanation either. When there's a big disparity in faction sizes, you have no chance of winning, and when there's a small disparity in faction sizes, you don't gain much by siding with the smaller faction. And as size differential between groups increases, the smaller faction's chance of success should drop much more quickly than the opportunities for status with the smaller faction should rise.

So I admit it. I'm stumped. What does Less Wrong think?

Comment author: pjeby 31 March 2009 07:42:01PM 3 points [-]

There the danger doesn't seem to be getting something that isn't the truth, the danger is stopping at something that's just true enough for a certain purpose, and no more.

Why is that bad?

And a seeker of truth seems less likely to get stuck there than a seeker of win - witness classical mechanics, which is still close enough to be useful for everything practical, versus relativity, which exists because Einstein wouldn't accept a theory which worked well enough but had a few little loose ends.

How has relativity made us better off? If you want to pursue truth because you like truth, that's great -- it's a "win" for you. But if you only need the truth to get to something else, it's not a win to add useless knowledge.

Are you sure that this isn't all about signaling being a truth-seeker? (i.e. "Truth-Seeking Isn't About The Truth")

After all, credibly signaling that you value the truth could make you a valuable ally, be considered a neutral judge, etc. etc. For these reasons, credibly valuing the truth above all else might be beneficial... for reasons not having anything to do with actually getting to the truth.

So, if you're saying we should seek truth just because it's the truth, and not because it brings practical benefit or pleasure or sends good signals, then what is the use of seeking truth?

Comment author: Yvain 31 March 2009 08:07:42PM 6 points [-]

Why is that bad?

It's not, if you know you're doing it.

Are you sure that this isn't all about signaling being a truth-seeker?

Pretty sure. If I wanted to signal, I'd be a lot more high-falutin about it. Actually, my comments do sound a bit high-falutin' (I was looking for a better word than "truth seeker", but couldn't find one) but that wasn't exactly what I wanted to express. The untangling-wires metaphor works a little better. Nominull's "I only seek to be right because I hate being wrong." works too. It's less of a "I vow to follow the pure light of Truth though it lead me to the very pits of Hell" and more of an "Aaargh, my brain feels so muddled right now, how do I clear this up?"

Also, this would be a terrible community to signal truth-seeking in, considering how entrenched the "rationality as win" metaphor is. As I mentioned in the hair example, I think a lot more people here are signaling a burning interest in real-world application than really have one.

So, if you're saying we should seek truth just because it's the truth, and not because it brings practical benefit or pleasure or sends good signals, then what is the use of seeking truth?

Um...this line of argument applies to everything, doesn't it? What is the use of seeking money, if it doesn't bring pleasure or send good signals? What is the use of seeking love, if it doesn't bring pleasure or send good signals? What is the use of seeking 'practical benefits', if they don't bring pleasure or send good signals?

Darned if I know. That's the way my utility function works. And it certainly is mediated by pleasure and good signals, but I prefer not to say it's about pleasure and good signals because I'd rather not be turned into orgasmium just yet.

Comment author: MichaelVassar 31 March 2009 05:03:03PM 4 points [-]

I'd sure appreciate exact info on the monetary cost/benefit breakdown for more expensively styled hair.
OTOH, isn't that an instance of trying to use rationality to win as well as one of trying to use great hair to win? Rationality will very rarely win directly.
It definitely won't win by failing to go meta and not telling us to spend some of our time on anything else, but if it looks like it's telling us that and we don't win it seems silly to blame it rather than ourselves for using it wrong or making it an idol.

Comment author: Yvain 31 March 2009 06:37:27PM 6 points [-]

I agree that it's rational to seek better hair if better hair leads to your goals. I'm trying to point out an inconsistency: that if you claim to be after success in business, and you spend a lot of time reading Less Wrong but very little time worrying over your hair, then either you're not being as rational as you think or you're not as focused on success in business as you think.

I further wonder if some people who read this will make a token attempt to consider getting nice haircuts, not because they're really after real-world success but because they want to be able to continue telling themselves credibly that they're really after real-world success.

Comment author: pjeby 31 March 2009 06:12:23PM 2 points [-]

If there's some other weird attractor state of beliefs that also fulfills those requirements, I guess I risk falling into it. But then again, so do you - such beliefs would have to predict experience as successfully as the truth, which means they would have to give you the same widget-making capacity as true beliefs.

There are plenty of things like this -- engineering models, heuristics, etc. You don't have to have a "true" map to have a "useful" map. An idealized right-angle, not-to-scale map of a city which nonetheless allowed you to logically navigate from point A to point B would be "useful" even if not "true" or "accurate" in certain senses.

Meanwhile, if you wait around for a "true" map, you're not going anywhere.

Comment author: Yvain 31 March 2009 06:31:41PM 3 points [-]

But such maps are only useful insofar as they are true. For example, the London Tube Map claims to be a useful representation of which stations are on which lines. It's useful in doing that because it is correct in its domain - every station it says is on the Piccadilly Line really is on the Piccadilly Line. It doesn't claim to accurately represent distance, and anyone who tried to use it to determine distances would quickly get some surprises.

There the danger doesn't seem to be getting something that isn't the truth, the danger is stopping at something that's just true enough for a certain purpose, and no more.

And a seeker of truth seems less likely to get stuck there than a seeker of win - witness classical mechanics, which is still close enough to be useful for everything practical, versus relativity, which exists because Einstein wouldn't accept a theory which worked well enough but had a few little loose ends.

Comment author: cousin_it 31 March 2009 02:51:13PM *  10 points [-]

Never mind widgets. I overemphasized business in the original post. Any kind of reality check will do. Newton had a reality check, what about us? Not many accurate predictions here. Choose any real-world metric that suits you - just don't degenerate into "what biases have you overcome today" soft-science bullshit.

I once joked that science has four levels, high to low: "this works", "this is true", "this sounds true", "this sounds neat". We here are still at number three, no?

Comment author: Yvain 31 March 2009 06:10:06PM *  6 points [-]

I think you have too limited a picture of what searching for truth entails, and that we don't have as great a difference between our views as you think.

Newton and Einstein used rationality to seek truth and bring unity to experience, not for practical results. But they were both smart enough to know they'd better check their results against experience, or they'd get the wrong answer and never be able to move further. If we're smart, we'll do the same, whether we're after truth or whatever.

Someone once said there were two kinds of rich people - those who really like having luxury goods, and those for whom money is just a way to keep score. The same could apply to rationalists; there are those who want some specific practical goal or predictive ability, and there are others for whom the ability to achieve practical goals or make predictions is a way to keep score. Einstein was happy to hear his theory successfully predicted the path of light during an eclipse, I'm sure, but not because he was in it for the eclipse-light-predicting.

Comment author: Yvain 31 March 2009 02:28:28PM *  15 points [-]

I wonder how far your observation is generalizable across all people. I would have predicted the opposite effect.

I tend to be much more willing to donate money to charities than to donate time. And I find this to be a general principle (ie I tend to pay my taxes without grumbling too much, but when some stupid government policy wastes my time, that's when I get angry and write to my congressperson).

Possible explanations: I grew up in a wealthy family, and/or I don't really actually spend money on anything beyond necessities because the library gives books out for free.

Informal poll: If asked to donate either one hour of free time, or your hourly wage, to a worthy charity that would receive equal benefit from either, which would you rather do? Disregard taxes being deducted from your wage and that sort of thing.

Comment author: cousin_it 31 March 2009 12:40:59PM *  14 points [-]

Truth isn't something you feel is true.

Programming abounds with theories that give their adherents a sense of clarity - for example the relational model of data, or object-oriented design, or REST. Funny that all those theories have active "evangelists". My experience with them all had a pattern of initial "wow" followed much later by a painful un-clarification, as contact with the real world brought out the theory's shortcomings. Curiously, my experience with Overcoming Bias has followed the same pattern: I bought into the ideas wholesale for the first months, then slowly grew disillusioned over about a year. Today it's all the same old same old. The idea of correcting cognitive biases just isn't as powerful by itself as we like to think. Hence my post.

Don't get stuck in the box an ideology outlines for you. It may be a good fit today but you'll hopefully grow.

Re PS: if rationality had no other real-world uses except business success, and in that single area hairstyle proved to be a greater contributor, then yes, I'd leave and never look back.

Comment author: Yvain 31 March 2009 02:11:08PM *  10 points [-]

And yet Newton didn't develop the theory of gravitation as part of his quest to make cheaper widgets. Nor did Einstein develop relativity because there was anything practical that classical mechanics couldn't do.

Looking for truth is not a fancy way of saying "looking for the first half-assed solution you can find so you can feel like you know something true and go back to what you were doing before."

The easiest way to get a bundle of beliefs that completely dissolve previously mysterious questions, don't contradict one another, don't contradict experience, don't contain sacred mysteries, and don't force you into dark side epistemology to maintain them - as far as I know the easiest way to get such beliefs is to believe things that are true.

If there's some other weird attractor state of beliefs that also fulfills those requirements, I guess I risk falling into it. But then again, so do you - such beliefs would have to predict experience as successfully as the truth, which means they would have to give you the same widget-making capacity as true beliefs.

Despite starting out at Less Wrong as a believer in the "rationalism helps you win!" school, the more I read the less I think going from the sort of person who's read all of Overcoming Bias to the sort of person who's read OB plus all of Less Wrong is going to grant you any hugely significant extra real-world-winning capacity. I should make a post on this sometime.

Comment author: Yvain 31 March 2009 12:04:41PM *  22 points [-]

All this stuff about "Something to Protect" and "Rationalists Should Win" is all nice and well. And Eliezer's point that grounding rationalism in some real-world need ensures that we don't enter affective death spirals around some particular less-than-optimal rational method, is well-taken.

But dammit, truth for the sake of truth is okay too!

I want to know how to become better at certain things, but that's not the main reason I'm on here. Have you ever had all the wires connected to your computer all tangled up, and it's not really making anything that much harder to use, but it just gets on your nerves until you have to stop what you're doing and spend however long it takes trying to untangle all of them and put them in nice little lines? That's how my brain feels all the time. Reading Overcoming Bias helped me untangle some of those wires.

Of course, if I didn't then take advantage of my new clarity to advance some of my other values, you'd have to wonder whether I'd really learned anything. But that wasn't the main reason I came here, and it's not the main reason I stay.

...and judging by how much time people here spend on Newcomb-like problems, then unless you have some really unusual day jobs I'm guessing I'm not alone.

PS: If you think rationality's important for succeeding at business, and I prove to you that great-looking hair is a greater contributor to success in business than knowledge of Bayescraft, would you stop reading Less Wrong and start reading hairstylist blogs? Would you recommend other people do so?

Comment author: gjm 30 March 2009 12:45:30AM 0 points [-]

You have (I assume) a copy-and-paste error: the phrase "The concept of culture" appears out of the blue in the middle of a paragraph, where it doesn't make any sense.

For what it's worth, the name "Nacirema" triggered my something-funny-going-on-here detectors right at the start, at which point it was all rather too obvious. Perhaps it's only for that reason that I found it a bit long.

(I liked it anyway. Were you inspired by this book?)

Comment author: Yvain 30 March 2009 12:58:19AM 1 point [-]

Copy-paste error fixed. Thank you. I'm afraid I wasn't inspired at all, though I wish I could take credit. The piece really is by an anthropologist named Horace Miner.

The book looks clever, though.

Comment author: Yvain 29 March 2009 10:48:21PM 4 points [-]

This whole koan guessing game is a bit silly. The whole point of koans is that it takes long months of meditation and a very special frame of mind to understand them. If a guy reading a blog could guess the answer after five minutes' thought, the koan wouldn't be worthwhile anyway (I am willing to grant Eliezer his fox koan, but even he only solved part of that one, and only with the help of centuries of Western philosophy and deep personal knowledge of the issues involved).

But I will hazard a guess as to Annoyance's interpretation. Annoyance wants us to consider mundane experience more. The monk was hoping Joshu would teach him something really exciting about Buddha or enlightenment or meditation. Instead, Joshu reminded him that after you eat, you've got to do the dishes. The monk realized that Zen wasn't about fancy theories of ultimate holy reality, but about living mundane experience as fully as possible. He also realized that his thoughts of enlightenment were distracting him from a task he should be carrying out with his full mind, and when he concentrated his full mind on the present moment, he achieved enlightenment.

Framing Effects in Anthropology

6 Post author: Yvain 29 March 2009 10:05PM

A large number of cognitive errors are grouped under "framing effects", the tendency of a fact to sound different when presented in different ways. Economists discuss framing effects in terms of changed decisions: for example, a patient will be more likely to agree to a treatment with a "ninety percent survival rate" than a "ten percent death rate", even though these are denotatively the same. Other social sciences use "framing" more broadly. For them, a frame is similar to a cultural filter through which we interpret and evaluate data.

Anthropologists are particularly wary of framing effects. The thought "primitive culture" immediately summons a set of associations - medicine men, chiefs, thatched huts, festivals, superstitions - that anthropologists risks interpreting new information about a tribe in light of what they think tribal cultures should be like. The problem is only compounded by the difficulty anthropologists have getting complete and accurate information from potentially reclusive societies.

One especially well-known anthropological work is Horace Miner's description of the Nacirema, a North American tribe centered around the northwest Chesapeake Bay area. He was especially interested in their purification customs, which he described as "an extreme of human behavior". Below the cut is Miner's essay, Body Ritual among the Nacirema. Do you think Miner is affected by a framing bias? Where does the bias manifest itself?

The anthropologist has become so familiar with the diversity of ways in which different peoples behave in similar situations that he is not apt to be surprised by even the most exotic customs. In fact, if all of the logically possible combinations of behavior have not been found somewhere in the world, he is apt to suspect that they must be present in some yet undescribed tribe.  This point has, in fact, been expressed with respect to clan organization by Murdock.  In this light, the magical beliefs and practices of the Nacirema present such unusual aspects that it seems desirable to describe them as an example of the extremes to which human behavior can go.

Nacirema culture is characterized by a highly developed economy which has evolved in a rich natural habitat. While much of the people's time is devoted to economic pursuits, a large part of the fruits of these labors and a considerable portion of the day are spent in ritual activity. The focus of this activity is the human body, the appearance and health of which loom as a dominant concern in the ethos of the people. While such a concern is certainly not unusual, its ceremonial aspects and associated philosophy are unique.

The fundamental belief underlying the whole system appears to be that the human body is ugly and that its natural tendency is to debility and disease. Incarcerated in such a body, man's only hope is to avert these characteristics through the use of the powerful influences of ritual and ceremony. Every household has one or more shrines devoted to this purpose. The more powerful individuals in the society have several shrines in their houses and, in fact, the opulence of a house is often referred to in terms of the number of such ritual centers it possesses. Most houses are of wattle and daub construction, but the shrine rooms of the more wealthy are walled with stone. Poorer families imitate the rich by applying pottery plaques to their shrine walls.  While each family has at least one such shrine, the rituals associated with it are not family ceremonies but are private and secret. The rites are normally only discussed with children, and then only during the period when they are being initiated into these mysteries. I was able, however, to establish sufficient rapport with the natives to examine these shrines and to have the rituals described to me.

The focal point of the shrine is a box or chest which is built into the wall. In this chest are kept the many charms and magical potions without which no native believes he could live. These preparations are secured from a variety of specialized practitioners. The most powerful of these are the medicine men, whose assistance must be rewarded with substantial gifts.  However, the medicine men do not provide the curative potions for their clients, but decide what the ingredients should be and then write them down in ancient and secret symbols. This writing is understood only by the medicine men and by the herbalists who, for another gift, provide the required charm.

The charm is not disposed of after it has served its purpose, but is placed in the charmbox of the household shrine. As these magical materials are specific for certain ills, and the real or imagined maladies of the people are many, the charm-box is usually full to overflowing. The magical packets are so numerous that people forget what their purposes were and fear to use them again. While the natives are very vague on this point, we can only assume that the idea in retaining all the old magical materials is that their presence in the charm-box, before which the body rituals are conducted, will in some way protect the worshipper.

Beneath the charm-box is a small font. Each day every member of the family, in succession, enters the shrine room, bows his head before the charm-box, mingles different sorts of holy water in the font, and proceeds with a brief rite of ablution. The holy waters are secured from the Water Temple of the community, where the priests conduct elaborate ceremonies to make the liquid ritually pure.

In the hierarchy of magical practitioners, and below the medicine men in prestige, are specialists whose designation is best translated "holy-mouth-men." The Nacirema have an almost pathological horror of and fascination with the mouth, the condition of which is believed to have a supernatural influence on all social relationships. Were it not for the rituals of the mouth, they believe that their teeth would fall out, their gums bleed, their jaws shrink, their friends desert them, and their lovers  reject them. They also believe that a strong relationship exists between oral and moral characteristics. For example, there is a ritual ablution of the mouth for children which is supposed to improve their moral fiber.

The daily body ritual performed by everyone includes a mouth-rite. Despite the fact that these people are so punctilious about care of the mouth, this rite involves a practice which strikes the uninitiated stranger as revolting. It was reported to me that the ritual consists of inserting a small bundle of hog hairs into the mouth, along with certain magical powders, and then moving the bundle in a highly formalized series of gestures.

In addition to the private mouth-rite, the people seek out a holy-mouth-man once or twice a year. These practitioners have an impressive set of paraphernalia, consisting of a variety of augers, awls, probes, and prods. The use of these objects in the exorcism of the evils of the mouth involves almost unbelievable ritual torture of the client. The holy-mouth-man open the clients mouth and, using the above mentioned tools, enlarges any holes which decay may have created in the teeth. Magical materials are put into these holes. If there age no naturally occurring holes in the teeth, large sections of one or more teeth are gouged out so that the supernatural substance can be applied. In the client's view, the purpose of these ministrations is to arrest decay and to draw friends. The extremely sacred and traditional character of the rite is evident in the fact that the natives return to the holy--mouth-men year after year, despite the fact  that their teeth continue to decay.

It is to be hoped that, when a thorough  study of the Nacirema is made, there will  be careful inquiry into the personality  structure of these people. One has but to  watch the gleam in the eye of a holy-  mouth-man, as he jabs an awl into an  exposed nerve, to suspect that a certain  amount of sadism is involved. If this can be  established, a very interesting pattern  emerges, for most of the population shows  definite masochistic tendencies. It was to  these that Professor Linton referred in discussing a distinctive part of the daily  body ritual which is performed only by  men. This part of the rite involves scraping  and lacerating the surface of the face with a  sharp instrument. Special women's rites are  performed only four times during each  lunar month, but what they lack in  frequency is made up in barbarity. As part  of this ceremony, women bake their heads  in small ovens for about an hour. The  theoretically interesting point is that what  seems to be a preponderantly masochistic  people have developed sadistic specialists.

The medicine men have an imposing  temple, or latipso, in every community of  any size. The more elaborate ceremonies  required to treat very sick patients can only  be performed at this temple. These ceremonies involve not only the thaumaturge  but a permanent group of vestal maidens  who move sedately about the temple  chambers in distinctive costume and head-dress.

The latipso ceremonies are so harsh that  it is phenomenal that a fair proportion of  the really sick natives who enter the temple ever recover. Small children whose indoctrination is still incomplete have been  known to resist attempts to take them to  the temple because "that is where you go to  die." Despite this fact, sick adults are not  only willing but eager to undergo the  protracted ritual purification, if they can  afford to do so. No matter how ill the  supplicant or how grave the emergency, the  guardians of many temples will not admit a  client if he cannot give a rich gift to the  custodian. Even after one has gained admission and survived the ceremonies, the  guardians will not permit the neophyte to  leave until he makes still another gift.

The supplicant entering the temple is  first stripped of all his or her clothes. In  everyday life the Nacirema avoids exposure  of his body and its natural functions.  Bathing and excretory acts are performed  only in the secrecy of the household shrine,  where they are ritualized as part of the  body-rites. Psychological shock results  from the fact that body secrecy is suddenly  lost upon entry into the latipso. A man,  whose own wife has never seen him in an  excretory act, suddenly finds himself naked  and assisted by a vestal maiden while he  performs his natural functions into a sacred  vessel. This sort of ceremonial treatment is  necessitated by the fact that the excreta are  used by a diviner to ascertain the course  and nature of the client's sickness. Female  clients, on the other hand, find their naked  bodies are subjected to the scrutiny,  manipulation and prodding of the medicine men.

Few supplicants in the temple are well  enough to do anything but lie on their  hard  beds. The daily ceremonies, like the rites of  the holy-mouth-men, involve discomfort  and torture. With ritual precision, the  vestals awaken their miserable charges each  dawn and roll them about on their beds of  pain while performing ablutions, in the  formal movements of which the maidens are highly trained. At other times they  insert magic wands in the supplicant's  mouth or force him to eat substances which  are supposed to be healing. From time to  time the medicine men come to their clients  and jab magically treated needles into their  flesh. The fact that these temple ceremonies  may not cure, and may even kill the  neophyte, in no way decreases the people's  faith in the medicine men.

There remains one other kind of  practitioner, known as a "listener." This  witchdoctor has the power to exorcise the devils that lodge in the heads of people who  have been bewitched. The Nacirema  believe that parents bewitch their own  children. Mothers are particularly suspected of putting a curse on children while  teaching them the secret body rituals. The  counter-magic of the witchdoctor is unusual in its lack of ritual. The patient simply tells the "listener" all his troubles and  fears, beginning with the earliest difficulties  he can remember. The memory displayed  by the Nacirerna in these exorcism sessions  is truly remarkable. It is not uncommon for  the patient to bemoan the rejection he felt  upon being weaned as a babe, and a few  individuals even see their troubles going  back to the traumatic effects of their own  birth.

In conclusion, mention must be made of  certain practices which have their base in  native esthetics but which depend upon the  pervasive aversion to the natural body and  its functions. There are ritual fasts to make  fat people thin and ceremonial feasts to  make thin people fat. Still other rites are  used to make women's breasts larger if they  are small, and smaller if they are large. General dissatisfaction with breast shape is symbolized in the fact that the ideal form is virtually outside the range of human variation. A few women afflicted with almost inhuman hyper-mamrnary development are so idolized that they make a   handsome living by simply going from village to village and permitting the natives to stare at them for a fee.

Reference has already been made to the fact that excretory functions are ritualized,   routinized, and relegated to secrecy. Natural reproductive functions are similarly distorted. Intercourse is taboo as a topic and scheduled as an act. Efforts are made to   avoid pregnancy by the use of magical   materials or by limiting intercourse to certain phases of the moon. Conception is   actually very infrequent. When pregnant, women dress so as to hide their condition.  Parturition takes place in secret, without   friends or relatives to assist, and the majority of women do not nurse their infants.

Our review of the ritual life of the Nacirema has certainly shown them to be a   magic-ridden people. It is hard to un-   derstand how they have managed to exist   so long under the burdens which they have   imposed upon themselves. But even such   exotic customs as these take on real   meaning when they are viewed with the insight provided by Malinowski when he  wrote:

"Looking from far and above, from our  high places of safety in the developed civilization, it is easy to see all the crudity and irrelevance of magic. But without its power and guidance early man could not   have mastered his practical difficulties as he has done, nor could man have advanced to the higher stages of civilization."

Now, spell "Nacirema" backwards and read it again.

The Zombie Preacher of Somerset

41 Post author: Yvain 28 March 2009 10:29PM

Related to: Zombies? Zombies!, Zombie Responses, Zombies: The Movie, The Apologist and the Revolutionary

All disabling accidents are tragic, but some are especially bitter. The high school sports star paralyzed in a car crash. The beautiful actress horribly disfigured in a fire. The pious preacher who loses his soul during a highway robbery.

As far as I know, this last one only happened once, but once was enough. Simon Browne was an early eighteenth century pastor of a large Dissident church. The community loved him for his deep faith and his remarkable intelligence, and his career seemed assured.

One fateful night in 1723, he was travelling from his birthplace in Somerset to his congregation in London when a highway robber accosted the coach carrying him and his friend. With quick reflexes and the element of surprise, Browne and his friend were able to disarm the startled highway robber and throw him to the ground. Browne tried to pin him down while the friend went for help, but in the heat of the moment he used excessive force and choked the man to death. This horrified the poor preacher, who was normally the sort never to hurt a fly.

Whether it was the shock, the guilt, or some unnoticed injury taken in the fight, something strange began to happen to Simon Browne. In his own words, he gradually became:

...perfectly empty of all thought, reflection, conscience, and consideration, entirely destitute of the knowledge of God and Christ, unable to look backward or forward, or inward or outward, having no conviction of sin or duty, no capacity of reviewing his conduct, and, in a word, without any principles of religion or even of reason, and without the common sentiments or affections of human nature, insensible even to the good things of life, incapable of tasting any present enjoyments, or expecting future ones...all body, without so much as the remembrance of the ruins of that mind I was once a tenant in...and the thinking being that was in me is, by a consumption continual, now wholly perished and come to nothing.

Simon Browne had become a p-zombie.

Needless to say, Browne's friends and congregation didn't believe him. Browne seemed as much in possession of his wits as ever. His writing, mostly on abstruse theological topics and ecumenialism, if anything accelerated. According to a friend:

What was most extraordinary in his case was this; that, excepting the single point I have mentioned, on which the distraction turned, his imagination was not only more lively, but his judgment was even improved. And it has been observed that, at the very time that he himself imagined he had no rational soul, he was so acute a disputant (his friends said) that he could reason as if he had two souls.

Despite everyone's insistence that he was fine, Simon Browne would have none of it. His soul had gone missing, and no one without a soul was qualified to lead a religious organization. Despite pleas to remain, he quit his job as pastor and retired to the country. After a brief period spent bemoaning his fate, he learned to take it in stride and began writing prodigously, authoring dictionaries, textbooks on grammars, essays on theology, and even several beautiful hymns still sung in churches today. Did his success convince him he was ensouled after all? No. He claimed:

...only an animal life, in common with brutes, so that though he retained the faculty of speaking in a manner that appeared rational to others, he had all the while no more notion of what he said than a parrot, being utterly divested of consciousness.

And, appreciating the absurdity of his conundrum, asked:

Who, by the most unreasonable and ill-founded conceit in the world, [could] have imagined that a thinking being could, for seven years together, live a stranger to its own powers, exercises, operations, and state?

Considering it pointless to exercise or to protect his own health, he died prematurely in his Somerset house in 1732. His friends mourned a potentially brilliant pastor driven to an early death by an inexplicable insanity.

But was his delusion really inexplicable?

David Berman is probably the top expert on the Simon Browne case, and the author of the only journal article dedicated specifically to the topic: Simon Browne: the soul-murdered theologian (other books that devote some space to Browne can be read here and here). I've been unable to access Berman's paper (if anyone can get it free, please send it to me) but I had the good fortune to be in his Philosophy of Mind class several years ago. If I remember correctly, Dr. Berman had a complex Freudian theory involving repression of erotic feelings. I don't remember enough to do it justice and I'm not going to try. But with all due respect to my former professor, I think he's barking up the wrong tree.

Simon Browne's problem seems strangely similar to neurological illness.

You remember anosognosia, when patients with left-arm paralysis thought their left arms were working just fine? Somatoparaphrenia is a closely related disorder. Your arm is working just fine, but you deny you have an arm at all. It must be someone else's. Some scientists link somatoparaphrenia to Body Integrity Identity Disorder, a condition in which people are desperate to amputate their working limbs for no apparent reason. BIID sufferers are sane enough to recognize that they do currently have a left arm, but it feels alien and unwelcome, and they want it gone.

(according to Wikipedia, one cure being investigated for BIID is squirting cold water in the patient's right ear...)

Somatoparaphrenia is an identity problem - people lose identity with their limbs. That arm might work, but it doesn't seem like it's working for me. Every other rational process remains intact in somatoparaphrenics. A somatoparaphrenic physicist could do quantum calculations while still insisting that someone else's leg was attached to his hip for some reason.

Cotard's Delusion is an even worse condition where the patient insists she is dead or nonexistent. Tantalizingly, patients with Cotard's occasionally use religious language, claiming to have been eternally damned or without a soul - a symptom shared by Simon Browne. Unlike anosognosia and somatoparaphrenia, it is not necessarily caused by stroke - all sorts of things, neurological or psychological, can bring it on. V. S. Ramachandran (yes, him again) theorizes that Cotard's may be a disconnect between certain recognition circuits and certain emotional circuits, preventing the patient from feeling an emotional connection with himself.

Browne reminds me also of "blindsight", the phenomenon where a patient is capable of seeing but not consciously aware of doing so. Ask a patient what she sees, and she'll swear she sees nothing - she is, after all, totally blind. Ask a patient to guess which of four quarters of the visual field a light is in, and she'll look at you like an idiot. How should she know? She's blind! Harass the patient until she finally guesses, and she'll get it right, at odds phenomenally greater than chance. Ask her how she knew, and she'll say it was a lucky guess.

Simon Browne sits somewhere in between all of these. Like the Cotard patient, he denied having a self, and considered himself eternally damned. Like the somatoparaphreniac, he completely lost identification with a certain part of himself (in this case, the mind!) and insisted it didn't exist while retaining the ability to use it and to reason accurately in other domains. And like the blindsight patient, he was able to process information at a level usually restricted to conscious experience without any awareness of doing so.

I don't know any diagnosis that exactly fits Browne's symptoms (Cotard's comes close but falls a little short). But the symptoms seem so reminiscent of neurological injury that I would be surprised if Dr. Berman's psychoanalysis was the full story.

So, what does Simon Browne add to the p-zombie debate?

Either nothing or everything. We can easily dismiss him as a complete nutcase, no more accurate in describing his mental states than a schizophrenic is accurate in describing his conversations with angels. Or we can come up with a neurological explanation in which he has conscious experience, but considers it alien to himself.

I acknowledge the possibility, but it rings hollow. Browne's friends were unanimous in describing him as rational and intelligent. And Browne himself was very clear that he had no mental experience whatsoever, not that he had some mental experience that didn't seem like his own.

But if we accepted Browne as mostly truthful, it demonstrates consciousness is not an inseparable byproduct of normal mental operation. It is possible to take consciousness, remove it, and have a p-zombie left. Not a perfect Chalmerian p-zombie - Browne made it very clear that he noticed and cared deeply about his loss of consciousness, and didn't go around claiming he was still fully aware or any nonsense like that - but a p-zombie nonetheless.

That is a heck of a conclusion to draw from one poorly studied case (there is rumored to be a second similar case, one Lewis Kennedy, but I can't find information on this one). However, Simon Browne at the very least deserves to be shelved alongside the other scarce and contradictory evidence on this topic. Let's give the poor preacher the last word:

God should still have left me the power of speech, [that] I may at last convince [you] that my case has not been a delusion of fancy, but the most tremendous reality.

 

Comment author: Yvain 29 March 2009 09:02:37PM 22 points [-]

I've often observed that my ability to think creatively disappears after spending enough time having the noncreative normal way ground into me.

For example, I remember my first day on a job I noticed a bunch of ways the company was doing things inefficiently and could be better. After doing it the company's way for a year or two, the company's system seemed so natural that it didn't seem like there was anything wrong with it. But when I remembered some of the things I'd told people that first day, they still seemed like good ideas, even though I was no longer able to spontaneously generate them anymore.

Likewise, I think being in a field for a long time etches the paradigm into your brain so deeply that it inhibits your ability to think outside of it.

I think this probably works alongside any changes that might happen simply due to age. I'd like to see a study comparing the creativity of old people who are just joining a new field, versus relatively young people who have been in the field their whole lives.

Comment author: Yvain 29 March 2009 08:52:31PM *  2 points [-]

I'm very glad to see you doing this. I was going to try something similar (see the discussion here ) but I got lazy and distracted and I probably wasn't up to the data processing tasks anyway.

I was throwing together a questionnaire here. You're welcome to look over it and take any questions you think are interesting.

The survey's goals were different from yours - it was more of an attempt to collect information on Less Wrong members - but enough of the questions look similar to yours that you might be interested in seeing the rest.

Comment author: cousin_it 29 March 2009 09:16:49AM *  1 point [-]

Thomas Ligotti's short horror story "The Shadow, The Darkness" describes a man becoming a p-zombie due to extreme pain caused by a stomach malady. He describes the transformation as losing his "fabricated personality" to become a "successful organism", and actually finds international success as a sculptor afterwards.

I believe the traditional mystical term for this experience is "ego death". Many seek it.

Comment author: Yvain 29 March 2009 03:25:09PM 3 points [-]

Without being an expert, I think ego-death is completely different. Something like an abolition of the desire to control and fit a narrative to experience, not an abolition of experience itself. And all the mystics say ego-death is followed by a superior form of experience, which clearly didn't happen to Browne.

Comment author: AndrewD 29 March 2009 10:45:36AM 2 points [-]

The problem being that its unclear that this materialistic notion of zombiehood does any justice to the Chalmerian notion of a zombie.

The notion that this could indeed qualify as a valid form of zombiehood arises if one views the priest's transformation as involving an absence of qualia; but there is no indication that the priest lacks qualia. The priest certainly appears to lack affect, but this is not the same as an absence of qualia. There are phenomenal properties associated with affective (emotional) states; but like all qualia, these would seem epiphenomenal, being neither necessary nor perhaps sufficient for the existence of affective states (which be characterised in (complex) functional terms).

To show that the priest was denuded of qualia, it would be necessary to show that, for example, when viewing a rose, the priest no longer had an occurent experience of phenomenal red. But, as noted by Andrew and Ciphergoth, if we assume a dualistic view of qualia, demonstrating this loss of qualia is just not possible. Zombies claim not to be zombies and therefore have no authority on their zombie-hood. By virtue of the same lack of empirical verifiability for zombie-hood, any claim by the priest to be a zombie would likewise lack authority.

An interesting further consequence is that, on some views (notably not the view taken by Chalmers), non-zombies (presumably, all of us) cannot authoritatively claim that they are actually conscious.

Comment author: Yvain 29 March 2009 03:19:24PM 11 points [-]

Part of the point I think Browne's case demonstrates is that most p-zombie discussions contains a false dichotomy between the materialist view that qualia are so tightly linked to normal human thought that it's impossible to have one without the other, and the epiphenomenalist view that qualia are so loosely linked to human thought that a human without qualia would be exactly the same.

Eliezer and the other materialists' attack on the epiphenomenalist position is that its assertion that humans would even talk about qualia if they lacked qualia is absurd. This is a good argument.

But Browne suggests a more middle-of-the-line position: that qualia are not necessary for normal human activity, but that it is still possible to notice their absence in exactly the way one would expect. This preserves the intuitive notion that qualia are different from simple processing, and lacks the vulnerability of Chalmers' theory.

Of course, it just makes what qualia actually are more mysterious than ever. But no one ever said that would be easy.

Comment author: PhilGoetz 28 March 2009 11:05:00PM *  14 points [-]

But if we accepted Browne as mostly truthful, it demonstrates consciousness is not an inseparable byproduct of normal mental operation. It is possible to take consciousness, remove it, and have a p-zombie left.

I think that if this happened, the p-zombie wouldn't claim to be a p-zombie. The p-zombie would claim to be just like you and me. So something different must have happened to Browne. (Also, he wasn't injured.)

I have the article. How can I send it to you?

Comment author: Yvain 28 March 2009 11:15:19PM 4 points [-]

Send to yvain314@hotmail.com , and thank you much.

Is there a word for a person without consciousness who functions perfectly normally except for correctly admitting he lacks consciousness? I thought p-zombie was broad enough to cover both that and Chalmers-type zombies.

The Zombie Preacher of Somerset

41 Post author: Yvain 28 March 2009 10:29PM

Related to: Zombies? Zombies!, Zombie Responses, Zombies: The Movie, The Apologist and the Revolutionary

All disabling accidents are tragic, but some are especially bitter. The high school sports star paralyzed in a car crash. The beautiful actress horribly disfigured in a fire. The pious preacher who loses his soul during a highway robbery.

As far as I know, this last one only happened once, but once was enough. Simon Browne was an early eighteenth century pastor of a large Dissident church. The community loved him for his deep faith and his remarkable intelligence, and his career seemed assured.

One fateful night in 1723, he was travelling from his birthplace in Somerset to his congregation in London when a highway robber accosted the coach carrying him and his friend. With quick reflexes and the element of surprise, Browne and his friend were able to disarm the startled highway robber and throw him to the ground. Browne tried to pin him down while the friend went for help, but in the heat of the moment he used excessive force and choked the man to death. This horrified the poor preacher, who was normally the sort never to hurt a fly.

Whether it was the shock, the guilt, or some unnoticed injury taken in the fight, something strange began to happen to Simon Browne. In his own words, he gradually became:

...perfectly empty of all thought, reflection, conscience, and consideration, entirely destitute of the knowledge of God and Christ, unable to look backward or forward, or inward or outward, having no conviction of sin or duty, no capacity of reviewing his conduct, and, in a word, without any principles of religion or even of reason, and without the common sentiments or affections of human nature, insensible even to the good things of life, incapable of tasting any present enjoyments, or expecting future ones...all body, without so much as the remembrance of the ruins of that mind I was once a tenant in...and the thinking being that was in me is, by a consumption continual, now wholly perished and come to nothing.

Simon Browne had become a p-zombie.

Needless to say, Browne's friends and congregation didn't believe him. Browne seemed as much in possession of his wits as ever. His writing, mostly on abstruse theological topics and ecumenialism, if anything accelerated. According to a friend:

What was most extraordinary in his case was this; that, excepting the single point I have mentioned, on which the distraction turned, his imagination was not only more lively, but his judgment was even improved. And it has been observed that, at the very time that he himself imagined he had no rational soul, he was so acute a disputant (his friends said) that he could reason as if he had two souls.

Despite everyone's insistence that he was fine, Simon Browne would have none of it. His soul had gone missing, and no one without a soul was qualified to lead a religious organization. Despite pleas to remain, he quit his job as pastor and retired to the country. After a brief period spent bemoaning his fate, he learned to take it in stride and began writing prodigously, authoring dictionaries, textbooks on grammars, essays on theology, and even several beautiful hymns still sung in churches today. Did his success convince him he was ensouled after all? No. He claimed:

...only an animal life, in common with brutes, so that though he retained the faculty of speaking in a manner that appeared rational to others, he had all the while no more notion of what he said than a parrot, being utterly divested of consciousness.

And, appreciating the absurdity of his conundrum, asked:

Who, by the most unreasonable and ill-founded conceit in the world, [could] have imagined that a thinking being could, for seven years together, live a stranger to its own powers, exercises, operations, and state?

Considering it pointless to exercise or to protect his own health, he died prematurely in his Somerset house in 1732. His friends mourned a potentially brilliant pastor driven to an early death by an inexplicable insanity.

But was his delusion really inexplicable?

David Berman is probably the top expert on the Simon Browne case, and the author of the only journal article dedicated specifically to the topic: Simon Browne: the soul-murdered theologian (other books that devote some space to Browne can be read here and here). I've been unable to access Berman's paper (if anyone can get it free, please send it to me) but I had the good fortune to be in his Philosophy of Mind class several years ago. If I remember correctly, Dr. Berman had a complex Freudian theory involving repression of erotic feelings. I don't remember enough to do it justice and I'm not going to try. But with all due respect to my former professor, I think he's barking up the wrong tree.

Simon Browne's problem seems strangely similar to neurological illness.

You remember anosognosia, when patients with left-arm paralysis thought their left arms were working just fine? Somatoparaphrenia is a closely related disorder. Your arm is working just fine, but you deny you have an arm at all. It must be someone else's. Some scientists link somatoparaphrenia to Body Integrity Identity Disorder, a condition in which people are desperate to amputate their working limbs for no apparent reason. BIID sufferers are sane enough to recognize that they do currently have a left arm, but it feels alien and unwelcome, and they want it gone.

(according to Wikipedia, one cure being investigated for BIID is squirting cold water in the patient's right ear...)

Somatoparaphrenia is an identity problem - people lose identity with their limbs. That arm might work, but it doesn't seem like it's working for me. Every other rational process remains intact in somatoparaphrenics. A somatoparaphrenic physicist could do quantum calculations while still insisting that someone else's leg was attached to his hip for some reason.

Cotard's Delusion is an even worse condition where the patient insists she is dead or nonexistent. Tantalizingly, patients with Cotard's occasionally use religious language, claiming to have been eternally damned or without a soul - a symptom shared by Simon Browne. Unlike anosognosia and somatoparaphrenia, it is not necessarily caused by stroke - all sorts of things, neurological or psychological, can bring it on. V. S. Ramachandran (yes, him again) theorizes that Cotard's may be a disconnect between certain recognition circuits and certain emotional circuits, preventing the patient from feeling an emotional connection with himself.

Browne reminds me also of "blindsight", the phenomenon where a patient is capable of seeing but not consciously aware of doing so. Ask a patient what she sees, and she'll swear she sees nothing - she is, after all, totally blind. Ask a patient to guess which of four quarters of the visual field a light is in, and she'll look at you like an idiot. How should she know? She's blind! Harass the patient until she finally guesses, and she'll get it right, at odds phenomenally greater than chance. Ask her how she knew, and she'll say it was a lucky guess.

Simon Browne sits somewhere in between all of these. Like the Cotard patient, he denied having a self, and considered himself eternally damned. Like the somatoparaphreniac, he completely lost identification with a certain part of himself (in this case, the mind!) and insisted it didn't exist while retaining the ability to use it and to reason accurately in other domains. And like the blindsight patient, he was able to process information at a level usually restricted to conscious experience without any awareness of doing so.

I don't know any diagnosis that exactly fits Browne's symptoms (Cotard's comes close but falls a little short). But the symptoms seem so reminiscent of neurological injury that I would be surprised if Dr. Berman's psychoanalysis was the full story.

So, what does Simon Browne add to the p-zombie debate?

Either nothing or everything. We can easily dismiss him as a complete nutcase, no more accurate in describing his mental states than a schizophrenic is accurate in describing his conversations with angels. Or we can come up with a neurological explanation in which he has conscious experience, but considers it alien to himself.

I acknowledge the possibility, but it rings hollow. Browne's friends were unanimous in describing him as rational and intelligent. And Browne himself was very clear that he had no mental experience whatsoever, not that he had some mental experience that didn't seem like his own.

But if we accepted Browne as mostly truthful, it demonstrates consciousness is not an inseparable byproduct of normal mental operation. It is possible to take consciousness, remove it, and have a p-zombie left. Not a perfect Chalmerian p-zombie - Browne made it very clear that he noticed and cared deeply about his loss of consciousness, and didn't go around claiming he was still fully aware or any nonsense like that - but a p-zombie nonetheless.

That is a heck of a conclusion to draw from one poorly studied case (there is rumored to be a second similar case, one Lewis Kennedy, but I can't find information on this one). However, Simon Browne at the very least deserves to be shelved alongside the other scarce and contradictory evidence on this topic. Let's give the poor preacher the last word:

God should still have left me the power of speech, [that] I may at last convince [you] that my case has not been a delusion of fancy, but the most tremendous reality.

 

In response to Church vs. Taskforce
Comment author: Yvain 28 March 2009 12:11:13PM *  7 points [-]

How is this significantly different from the Lions Club and Kiwanis, crossed with the local atheist organization?

I see how it's more rationalist-oriented than the Kiwanis, and more service-oriented than the Atheist Club. And they could probably get more charitable value for money by focusing on high-utility causes - if the rationalists were high-level enough, which the sort of people who respond to "rationalist club" ads might not be. But does "altruist rationalists" correspond to such a significant cluster in personspace that they need their own club? And is this just "we should start a fraternal organization"?

These clubs are interesting and do some good work, but I don't hear people speaking of them in the same breath as religion (except maybe when they get mystical, like the Freemasons).

Comment author: Yvain 28 March 2009 11:51:11AM *  8 points [-]

This is important stuff. Compare this post to Tom's observation that almost all Less Wrong readers are white 20-something males with a science degree.

Comment author: Marshall 28 March 2009 07:36:23AM 6 points [-]

Yvain -

i) you have inherited the longwindedness of Eliezer. I wonder if this could be elevated to a fallacy? Let's just raise our hands (on second thoughts not).

ii) the Steve Jobs aphorism is an aphorism - compact, pithy, witty. Like a good metaphor it surprises and sends the mind on a search. The reader himself finds meaning and defies common logic and possible fallacies. Metaphors are dangerous.

iii) it is my strong impression that rational rationalists hate metaphors. Metaphors are not based on evidence or deconstruction but more a pluralistic synthesis mode. Intuitive emergence (oops - bad word) as a tool for irrational rationalists.

iv) the expressionn "Dark Arts" is quickly becoming a weapon of destruction. While I have no wish to do evil things and incant magic rituals I suggest real rationality (and not just the rationality as practised by white male english-speaking twentysomethings with degrees in maths or science) involves other modes of cognition and have more shades of colour than LW allows.

Comment author: Yvain 28 March 2009 11:37:48AM 4 points [-]

I know I'm long-winded, and apologize. I'm busy, and as Pascal said, "This letter is too long, because I lacked the time to make it shorter."

I like metaphors. They're useful for illuminating things. I can go on and on about how failing the IAT doesn't make you racist because you can hold beliefs on two different levels, or I can say "It's like running away from a haunted house even if you don't believe in ghosts." I haven't formalized my intuitions on which metaphors are or aren't good yet, but I should.

IV seems plausible, but I can't think of exactly how. Sounds like it deserves its own top-level post.

Comment author: Technologos 28 March 2009 05:37:50AM 19 points [-]

I disagree with your conclusion on the grounds that I think you're interpreting the passage in a different way than the author intended it.

My interpretation is more along the lines of "Steve Jobs, on seeing the profound economic destitution in the East, examined his beliefs about human utility functions. Finding that Western values/practices seemed to promote utility better than Eastern values/practices in developing countries (and perhaps that they do even in developed countries), he decided that the East should adopt the West's values/practices."

I loved the dissection into multiple biases, and I agree with your dissection based on your reading of the passage, but I think the author is saying something different and more in line with the prevalent beliefs on Less Wrong.

Comment author: Yvain 28 March 2009 11:23:55AM *  15 points [-]

Yes and no. This is a reasonable defense of Steve Jobs, but not a reasonable defense of using the statement as a reply to Crowley's theory of yoga. The first conclusion of the statement - that India needs more Western values - is okay. The second conclusion - that therefore, the West can't learn from Indian "spiritual wisdom" - doesn't follow. To reply to Crowley's theory, it needs to prove the second conclusion.

Defense Against The Dark Arts: Case Study #1

100 Post author: Yvain 28 March 2009 02:31AM

Related to: The Power of Positivist Thinking, On Seeking a Shortening of the Way, Crowley on Religious Experience

Annoyance wants us to stop talking about fancy techniques and get back to basics. I disagree with the philosophy behind his statement, but the principle is sound. In many areas of life - I'm thinking mostly of sports, but not for lack of alternatives - mastery of the basics beats poorly-grounded fancy techniques every time.

One basic of rationality is paying close attention to an argument. Dissecting it to avoid rhetorical tricks, hidden fallacies, and other Dark Arts.  I've been working on this for years, and I still fall short on a regular basis.

Medical educators have started emphasizing case studies in their curricula. Instead of studying arcane principles of disease, student doctors cooperate to analyze a particular patient in detail, ennumerate the principles needed to diagnose her illness, and pay special attention to any errors the patients' doctors made during the treatment. The cases may be rare tropical infections, but they're more often the same everyday diseases common in the general population, forcing the student doctors to always keep the basics in mind. We could do with a tradition of case studies in rationality, though we'd need safeguards to prevent degeneration into political discussion.

Case studies in medicine are most interesting when all the student doctors disagree with each other. To that end, I've chosen as the first case a statement that received sixteen upvotes on Less Wrong, maybe the highest I've ever seen for a comment. I don't mean to insult or embarass everyone who liked it. I liked it too. My cursor was already hovering above the "Vote Up" button by the time I starting having second thoughts. But it deserves dissection, and its popularity gives me a ready response when someone says this material is too basic for 'master rationalists' like ourselves:

In his youth, Steve Jobs went to India to be enlightened. After seeing that the nation claiming to be the source of this great spiritual knowledge was full of hunger, ignorance, squalor, poverty, prejudice, and disease, he came back and said that the East should look to the West for enlightenment.

This anecdote is short, witty, flattering, and utterly opaque to reason. It bears all the hallmarks of the Dark Arts.


I admit I am not a disinterested party here. The statement was in response to my claim that Indian yoga was a successful technique for inducing exotic and occasionally useful mental states. I don't like being told I'm wrong any more than anyone else does. But here I don't think I am. I see at least five fallacies.

First, a hidden assumption: if A is superior to B, A cannot learn anything from B. This assumption is clearly false. I know brilliant scientists whose spelling is atrocious. I acknowledge that these people are much smarter than I am, but I still correct their spelling. Anyone who said "Dr. A should not be learning spelling from Yvain, Yvain should be learning science from Dr. A" would be missing the point. If Dr. A wants to learn spelling, he might as well learn it from me. And best of all if we both learn from each other!

A related fallacy would be that Dr. A is so much smarter than the rest of us that he should not care about spelling. But if spelling is important to his work (perhaps he's writing a journal article) he needs to do everything he can to perfect it. If he could spell correctly, he would be even further ahead of the rest of us than he already is. The goal isn't to become a bit better than your peers and then rest on your laurels. The goal is to become as skilled as necessary.

The error is an interesting variant of the halo effect: that anyone superior at most things must be superior at all things.

Second, the statement assumes that India is a single monolithic entity with or without spiritual wisdom. But even the most gushing Orientalist would not study at the feet of a call-centre worker in Bangalore. Whatever spiritual wisdom may exist in India, it will be believed by a small fraction of Indian religions, be practiced by a small fraction of the believers, and be mastered by a small number of the practioners. And if Crowley is to be believed, it will be understood by a small fraction of the masters.

Compare the question: if America is so good at science, why does it have so many creationists? Well, because the people who are good at science aren't the same ones believing in creationism, that's why. And the people who are good at science don't have enough power in society to do anything about the creationism issue. This does not reflect poorly on the truth-value of scientific theories discovered by Americans.

I'm not one of those fallacy classification nuts, but for completeness' sake, this is a fallacy of composition.

Third, the statement assumes that spiritual wisdom makes people less poor and squalid. The converse of this statement certainly isn't true - being rich and sanitary doesn't give you any spiritual value, as large segments of western civilization have spent the past three hundred years amply demonstrating. People commonly interpret spiritual wisdom as conferring a disdain for material goods. So we wouldn't necessarily expect to see a lot of material well-being in a spiritually wise society.

Part of this is a problem with the definition of "spiritual wisdom". It can mean anything from "being a moral person who cares about others" to "being wise and able to make good decisions" to "having mastery of certain mental techniques that produce awe-inspiring experiences" Under the first and second definition, a spiritually attained country should be a nice place to live. Under the third definition, not so much. Crowley endorses the third definition, and believes that most spiritually wise people dismiss the mundane world as unworthy of their attention anyway. But this contradicts our usual intuitions about "spirituality" and "wisdom".

This is a failure of definition, and it's why I prefer "high level of mystical attainment" to "spiritually wise" when discussing Crowley's theories.

Fourth, this is hardly a controlled experiment. India is historically, geographically, racially, religiously, climatologically, and culturally different from the West. Attributing a certain failure to religious causes alone is highly dubious. In fact, when we think about it for a while, cramming a billion plus people into a sweltering malarial flood plain, dividing them evenly between two religions that hate each other's guts, then splitting off the northwest corner and turning it into a large populous nuclear-armed arch-enemy that declares war on them every couple of decades is probably not a recipe for success no matter what your spirituality. All we can say for certain is that India's spirituality is not sufficiently wonderful to overcome its other disadvantages.

People who like Latin call this cum hoc ergo propter hoc.

Fifth, this equivocates the heck out of the word "enlightenment". Compare "enlightenment" meaning the set of rational values associated with Newton, Descartes, and Hume, to "enlightenment", meaning gaining important knowledge, to "enlightenment", meaning achieving a state of nirvana free from worldly desire. The West is the acknowledged master of the first definition, and India the acknowledged master of the third definition. The anecdote's claim seems to be that since the West is the acknowledged master of the first type of enlightenment, and could teach India some useful things about politics and economics in the second sense of enlightenment, India can't teach the West about the third sense of enlightenment...which would make sense, if the types of enlightenment were at all related instead of being three different things called by the same name.

This is a fallacy of equivocation.

Just because I can point out a few fallacies in a statement doesn't make it worthless. Spiritual wisdom doesn't always correlate with decent living conditions, but the lack of decent living conditions is some evidence against the presence of spiritual wisdom. Likewise, a country's success or failure doesn't always depend on its religion, but religion is one of many contributing factors that does make a difference.

Still, five fallacies is a lot for a two sentence anecdote.

I don't think we all liked this anecdote so much because of whatever tiny core of usefulness managed to withstand those five fallacies. I think we liked it because it makes a good way to shut up hippies.

Hippies are always going on about how superior India is to the West in every way because of its "spirituality" and such, and how many problems are caused by "spiritually bankrupt" Western science. And here we are, people who quite like Western science, rolling our eyes at how stupid the hippie is being. Doesn't she realize that Western science gives her all of the comforts that make her life bearable, from drinkable water to lice-free clothing? And this anecdote - it strikes a blow for our team. It makes us feel good. We don't need to look to India for enlightenment! India should look to us! Take that, hippie!

But reversed stupidity is not intelligence. Just because the hippie is wrong about India, doesn't mean we have to be wrong in the opposite direction. It might be useful to share it with this hypothetical hippie, just to start her thinking. But it's not something we can seriously endorse.

Nor do I accept the defense that it was not specifically posted with the conclusion "Therefore, ignore Crowley's views on yoga." Merely placing it directly below an article on enlightenment from India is a declaration of war and a hijack attempt on the train of thought. Saying "I hear people of African descent have a higher violent crime rate" is not a neutral act when spoken right before a job interview with a black person.

Defense Against the Dark Arts needs to become total and automatic, because it is the foundation upon which the complicated rationalist techniques are built. There's no point studying some complex Bayesian evidence-summing manuever that could determine the expected utility of studying yoga if an anecdote about Steve Jobs can keep you from even considering it.

How do you know you have mastered this art? When the statements

In his youth, Steve Jobs went to India to be enlightened. After seeing that the nation claiming to be the source of this great spiritual knowledge was full of hunger, ignorance, squalor, poverty, prejudice, and disease, he came back and said that the East should look to the West for enlightenment.

and

For complex historical reasons, the average Westerner is richer than the average Indian. Therefore, there is minimal possibility that any Indian people ever discovered interesting mental techniques.

sound exactly alike.

Comment author: Nick_Tarleton 28 March 2009 12:19:34AM 1 point [-]

I'm still confused. Which general principle?

Comment author: Yvain 28 March 2009 12:40:57AM 1 point [-]

Outside view less biased than inside view.

Comment author: Yvain 27 March 2009 10:52:28PM *  1 point [-]

Since my original plan to avoid sticking my neck out by signing up only after a lot of other people did has a certain easily perceptible flaw...

...25 dollar one-time commitment to the charity chosen. Reserve the right to retract commitment if the system breaks down for some reason, ie obvious joke votes. Can't afford any more as I am a student, but hopefully this will prime the pump a little.

EDIT: In response to Eliezer's request, I retract this commitment, possibly in favor of whatever Eliezer's project is.

Comment author: Yvain 27 March 2009 08:25:05PM 21 points [-]

I called it an applause light last time because it makes you sound responsible, mature, and Deeply Wise without containing usable advice. I'd retract this if you could give examples of what in particular we should be doing.

If you consider mystical methods unlikely to bear fruit, and worse than a maximum ignorance prior, say it outright and justify it. But I can't tell if you're talking only about mystical "techniques" or also about psychological and Bayesian "techniques", and you think you're making a positive point instead of a negative one. I just can't figure out what that positive point is.

Yes, if there's something we know for sure increases rationality, we should be spending more time doing it instead of brainstorming new techniques. But first, there aren't many such things, and second, our inability to do them as much as we'd like is the akrasia complaint, which we've already flagged as something we need to work on.

Compare alcoholism. The mundane solution to alcoholism is to say "Just stop drinking so much" - this seems in keeping with your diet metaphor. This mundane solution very rarely works, thanks to a particularly nasty form of akrasia. The Alcoholics Anonymous program, various anti-alcoholism drugs, and other "gimmicks" are much more effective. We need techniques for an Irrationalists Anonymous program.

Everyone in the world is already having a lot of mundane experience, and it doesn't seem to have helped them much. We better do something different...and not just take more baths. Not necessarily some mystical ritual. It could just be learning a little more Bayesian math, studying lists of fallacies, or going to a philosophy class.

If your candidate for "do something different" is "pay more attention to mundane experience", then you need to define specific ways we can do that. If you literally just mean we should consciously try to elevate the level of mental attention with which we attend to daily tasks, then that's Zen. Hard Zen. Back during my Zen phase, I used to try this. Even a few minutes were unbearably difficult. It may be a valuable technique, but if it's really what you mean it needs more respect and rigor than you give it here.

If you want to continue this post as a series, please post some concrete examples of what we should be doing differently.

Comment author: cousin_it 27 March 2009 02:50:21PM *  2 points [-]

Yvain, what's the website? I'm not skilled enough to find it from those clues. Asking because I outlined the exact same idea in a blog post in Russian about a year ago, thinking it was original, and now am curious to see the implementation.

Comment author: Yvain 27 March 2009 02:59:40PM 3 points [-]

Hmm...got it bookmarked somewhere...ah! http://www.stickk.com/

Comment author: SoullessAutomaton 27 March 2009 02:18:40PM *  6 points [-]

Nonsense is not the same thing as falsehoods.

For comparison, I am happy to dismiss as true nonsense, and give no further consideration to, the significance of nature's 4-day simultaneous harmonic time cube.

Comment author: Yvain 27 March 2009 02:54:16PM 5 points [-]

You are stupid and evil, have been scammed by criminal educators you one-ist anti-intelligent fool.

Comment author: SoullessAutomaton 27 March 2009 02:10:30PM 4 points [-]

Even things that are untrue may provide information about the mental experiences of the person who believes them.

Religion and mysticism are far too common to simply dismiss as nonsense; it behooves those of us who reject them fundamentally to understand better their hold on the minds of others.

Comment author: Yvain 27 March 2009 02:51:43PM 4 points [-]

I agree with you, but we still can't go posting every bit of falsehood or New Age flimflam on Less Wrong just because there's probably something interesting behind it. A general article on the mental phenomena that underlie astrology would be interesting, but an astrologer's article on what it means for the moon to be in Aquarius, presented without comment, would not be.

Comment author: ciphergoth 27 March 2009 01:43:53PM 6 points [-]

Yes, we know that churches and cults thrive by exploiting well-understood cognitive biases, but you're sort of sidestepping the central thrust of what EY is getting at in this, which AFAICT is simply:

Isn't there some way we could make use of the power of collective action because it's actually a good idea, rather than relying on cognitive bias to cohere us? Rather than hanging onto the biases that bring us together, couldn't we get there by fighting the biases that keep us apart?

Comment author: Yvain 27 March 2009 02:26:44PM *  7 points [-]

No, I'm not saying they thrive by bias, exactly, or at least not the simple kind of bias. They thrive by having a hierarchy and being official. They thrive because they've made a commitment.

Consider marriage. In an ideal world, two people would stay monogamous purely because they loved each other. In reality, that monogamy is going to be tested, and there's going to be some point at which they don't want to keep it. When they're rational, they know the best thing for their future and their children is to stay together, but they realize that they might be too short-sighted to do so later. So they use the institution of marriage to make it socially, financially, and theologically impossible for them to split up later. It's the present self binding potentially irrational future selves. Not only is it not a bias, but if it's done right it's an antidote to bias.

There's that one website, whatsitsname, where you send them money and a resolution. Maybe it's "I will go to the gym every day for a month", and you send them $100. At the end of the month, if you went to the gym every day, they send your money back; if you didn't, they keep it. I wouldn't say you were biased into going to the gym, I'd say you discovered a clever technique to make you do it.

Organizations, at least the ones you join voluntarily, are another clever technique for causing that kind of commitment. And yeah, a lot of the techniques they use to do it, like the initiation ceremonies, are biases. But I don't consider biases that smart people invoke voluntarily to control their akrasia to always be great evils.

Comment author: Nick_Tarleton 27 March 2009 01:14:42AM 11 points [-]

Yes, I feel properly guilty posting this on a rationalism site

Um... why is guilt proper here? Rationalists win, Bayesians condition on all evidence, and so on.

Comment author: Yvain 27 March 2009 01:57:45PM *  2 points [-]

I believe this essay, despite being written by a mystic and involving mystical theories and little formal scientific evidence, is likely true and of value to rationalists.

But every crank or New Ager believes that their pet theory, despite being mystical and lacking scientific evidence, is likely true and of value to rationalists. So from the Inside View, I think the essay is valuable, but from the Outside View I'm forced to admit it might not be.

I chose to post it after people reacted positively to the comments I made based on it, but I still feel uncomfortable transgressing the general principle, hence the guilt.

Comment author: Yvain 27 March 2009 01:32:02PM *  9 points [-]

The reason Catholics are better organized than humanists is that they're official, communal, and hierarchical and we're not. The reason cults are better organized than Catholics is that they're even more official, communal, and hierarchical.

If the Pope says "Donate ten percent of your money to me," then there's an expectation that ordinary Catholics will obey. They've committed to following what the Pope says.

If you, Eliezer, posted on this forum "Please donate ten percent of your money to the Institute That Must Not Be Named", well...actually, I don't know what would happen. A few rare people might do it to signal that we liked you. But although we often follow you, we are not your followers. We haven't made a committment to you. We associate with you as long as it's convenient for us, but as soon as it stops being convenient, we'll wander off.

If you really want to get an infrastructure as powerful as the Catholic Church, you need to ask us to officially swear loyalty to you and start publically self-identifying as Rationalists with a capital R (the capital letter is very important!) You need to put us through some painful initiation ritual, so we feel a commitment to stick around even when the going gets tough. You need to make us publically profess how great Rationalism is to all our friends enough times that it would be a major social embarrassment to get kicked out for not obeying you enough. You need to establish a norm that following Eliezer's requests is so completely expected it would be strange to refuse and we'd be going against all our friends. And then you need to keep telling us about how much better off we are as capital-R Rationalists than as members of the boring old general public. Then you can start ordering us to donate ten percent of our income and expect Pope-level compliance rates.

The cultists do all of this, and the Catholics try but generally fail, which is why many Catholics don't listen to the Pope nearly as much as atheists think. If you didn't want to go quite this far, even making us pay $5 for a (physical, laminated, colorful) Less Wrong membership card would probably make a difference. Once we did that, we'd be members of something, instead of people who came to a blog every so often to discuss an interest. The brain cares a lot about this sort of thing.

[edit: better explanation below in response to ciphergoth]

Comment author: Yvain 26 March 2009 11:07:47PM *  11 points [-]

I'd say my problem was less that my mind wasn't designed for thinking than that my school wasn't.

There's a built-in assumption here that school is about developing thinking skills in the first place, whereas as far as I can tell it serves to dull them. If you end up with a more creative answer than the teacher, you're told to do it the "right" way. I remember inventing a faster way to do certain math problems, and being marked wrong on tests because I wasn't doing in the way I'd been taught. Most children naturally seek out knowledge - not just creative knowledge, but the "boring old facts" school teachers are so afraid of. School quashes this in the most effective way possible.

The main principle I take from this article (challenge your students in interesting ways) is pretty good, assuming an educational system completely different from the real one. But, like a lot of educational principles, it requires a lot of intelligence to apply intelligently, and most teachers either aren't smart enough or don't have the time.

Time and time again, I have seen teachers replace the substance of these techniques with the symbol, and ask "edgy" or "creative" sounding questions that annoy students and condescend to them while not teaching them anything substantive. I know because I lived through about a decade of it.

As for his "the mind is not designed for thinking" line, I call ADBOC.

[before anyone asks: I'm not insulting teachers. I spent two years as a teacher myself. I'm just saying some of them aren't up to the job, and the ones who are get bogged down in a system that keeps them from doing much good. Also, I reserve the right to dismiss this all later as an unjustified rant against basically sound cognitive science, but in my defense the full title of the article was about as condescending and offensive to good sense as could possibly be imagined.]

Crowley on Religious Experience

36 Post author: Yvain 26 March 2009 10:59PM

Reply to: The Sacred Mundane, BHTV: Yudkowsky vs. Frank on "Religious Experience"

Edward Crowley was a man of many talents. He studied chemistry at Cambridge - a period to which he later attributed his skeptical scientific outlook - but he soon abandoned the idea of a career in science and turned to his other passions. For a while he played competitive chess at the national level. He took to mountain-climbing, and became one of the early 20th century's premier mountaineers, co-leading the first expedition to attempt K2 in the Himalayas. He also enjoyed writing poetry and travelling the world, making it as far as Nepal and Burma in an era when steamship was still the fastest mode of transportation and British colonialism was still a thin veneer over dangerous and poorly-explored areas.

But his real interest was mysticism. He travelled to Sri Lanka, where he studied meditation and yoga under some of the great Hindu yogis. After spending several years there, he achieved a state of mystical attainment the Hindus call dhyana, and set about trying to describe and promote yoga to the West.

He was not the first person to make the attempt, but he was certainly the most interesting. Although his parents were religious fanatics and his father a fundamentalist preacher, he himself had been an atheist since childhood, and he considered the vast majority of yoga to be superstitious claptrap. He set about eliminating all the gods and chants and taboos and mysterian language, ending up with a short system of what he considered empirically validated principles for gaining enlightenment in the most efficient possible way.

Reading Crowley's essay on mysticism and yoga at age seventeen rewrote my view of religion. I had always wondered about eastern religions like Buddhism and Hinduism, which seemed to have some underlying truth to all their talk of "enlightenment" and "meditation" but which seemed too vague and mysterious for my liking. Crowley stripped the mystery away in one fell swoop.

When listening to Eliezer debate Adam Frank on "religious experience", I was disappointed but not surprised to hear just how little they had to say. Even Frank, who was fascinated enough to write a book about it, considered it little more than a sense that something was inspiring or especially impressive. I quoted a bit of Crowley's essay on the thread, and people seemed to like it and want to know more.

But I am very reluctant to share, and do so now only after being specifically requested by a few people. You see, I have been trying to paint a sympathetic view of Crowley over the past few paragraphs. With the unsympathetic view you are familiar already. Under his nickname "Aleister", he wrote some of history's most influential occultist works. Even in this domain, he held himself to a high rationalist standard, recording that he tested each spell and ritual beforehand and passed on only the ones that actually worked as advertised.

...I don't know what that means either. Either he was one of those psychopaths gifted with the ability to lie perfectly and absolutely, or a psychotic genius able to induce hallucinations in himself at will. Crowley himself occasionally endorsed this latter explanation, but after pondering it a while decided he didn't care. The important thing, he wrote, was to determine what techniques produced what results. After that, the philosophers could determine whether they were physical or mentally mediated. Besides, he said, the entities he summoned were so different from himself that if they represented faculties of his mind, they were ones to which he had no conscious access.

My point is that I am going to link you to Crowley's essay on mysticism, yoga, and religious experience, and that you might get more out of it if you tried to avoid any bias upon seeing the name "Aleister Crowley" on the title page. Yes, I feel properly guilty posting this on a rationalism site, but if we're going to talk about religious experience we might as well listen to the people who have had some.

Although it is Less Wrong tradition to rewrite a theory rather than simply link to it, it would be inappropriate in this case. Getting Crowley filtered would be like having someone summarize Godel, Escher Bach to you - you might learn a few things, but you'd lose the chance to enjoy the superb writing. It's a long essay, but not so long you can't read it in one sitting. Even just reading the Preface gives an idea of the theory. Without further ado: Crowley on Religious Experience.

I post this essay to clarify why I believe three things. First, that both Eliezer and Adam miss the point of religious experience. Second, that certain seemingly supernatural or silly beliefs can be more reasonable than they appear (see for example Crowley's explanation of religious laws on "virtue" and "purity"). Third, that some mystics'  work is of sufficient relevance to rationalists to be worth study.

Fight Biases, or Route Around Them?

25 Post author: Yvain 25 March 2009 10:23PM

Continuation of: The Implicit Association Test
Response to: 3 Levels of Rationality Verification

I've not yet seen it pointed out before that we use "bias" to mean two different things.

Sometimes we use "bias" to mean a hard-coded cognitive process that results in faulty beliefs. Take as examples the in-group bias, the recall bias, the bad guy bias, and various other things discovered by Tversky and Kahneman.

Other times, we use "bias" to mean a specific faulty belief generated by such a process, especially one that itself results in other faulty beliefs. For example, Jews are sometimes accused of having a pro-Israel bias. By this we mean that they have a higher opinion of Israel than the evidence justifies; this is a specific belief created by the in-group bias. This belief may itself generate other faulty beliefs; for example, they may have a more negative opinion of Palestinians than the evidence justifies. It is both the effect of a bias, and the cause of other biases.

Let's be clear about this "more than the evidence justifies" bit. Hating Hitler doesn't mean you're biased against Hitler. Likewise, having a belief about a particular ethnic group doesn't mean you're biased for or against them. My Asian friends hate it when people sheepishly admit in a guilty whisper that they've heard Asians are good at academics. Asians are good at academics. Just say "55% chance an average Asian has a GPA above the American population mean" and leave it at that. This is one of Tetlock's critiques of the Implicit Association Test, and it's a good one. I'd probably link Asians to high achievement on an IAT, but it wouldn't be a bias or anything to get upset about.

And let's also be clear about this faulty belief thing. You don't have to believe something for it to be a belief; consider again the skeptic who flees the haunted house. She claims she doesn't belief in ghosts, and she's telling the truth one hundred percent. She's still going to be influenced by her belief in ghosts. She's not secretly supernaturalist any more than someone who gets "strongly biased" on the IAT is secretly racist. But she needs to know she's still going to run screaming from haunted houses, and IAT-takers should be aware they're still probably going to discriminate against black people in some tiny imperceptible way.

Okay, back to the example. So the President appoints Isaac, a synagogue-going Jew, as the new Middle East peace envoy. Due to some amazing breakthrough in the region, both the Israelis and Palestinians agree to accept whatever plan Isaac develops. Isaac's only job is to decide what long-term plan is best for both sides. And he's a good man: he has an honest desire to choose the maximum-utility solution.

Isaac legitimately worries that he has a bias for the Israelis and against the Palestinians. How can he test the hypothesis? He can take a hypothetical souped-up version of the Implicit Association Test1. He finds that yes, he has a strong pro-Israel anti-Palestine bias. Now what does he do?

He can try to route around the bias. This is the approach implicitly endorsed by Overcoming Bias and by rationalism in general. He can take the Outside View and look at successful approaches in other world conflicts. He can use some objective metric to calculate the utility of everything in Israel, and check to make sure neither group is getting an amount disproportionate to their numbers. He can open a prediction market on metrics of success, and implement whatever policies trades at the highest value. All of these will probably improve Isaac's solution a fair bit. But none of them are perfect. In the end, Isaac's the one who has to make a decision that will be underdetermined by all these clever methods, and Isaac is still biased against the Palestinians.

Or he can try to fight the bias.

Diversity workshops try to fight biases directly . These don't work, and that's no surprise. Diversity workshops are telling you, on a conscious level, that minorities really are great people, aren't they? Well, yes. On a conscious level, you already believe that. Isaac already knows, on a conscious level, that the Palestinians deserve a fair solution that protects their interests just as much as the Israelis do. A diversity workshop would be a flashy video in which a condescending narrator explains that point again and again.

We don't have a lot of literature on what does work here, but I predict a few things would help. Make some Palestinian friends, to build mental connections between Palestinians and positive feelings. Learn to distinguish between Palestinian faces. Read works of fiction with sympathetic Palestinian characters. I would say "live in Palestine" but by all accounts Palestine is a pretty grim place; he might do better to live in a Palestinian community in America for a while.

Those techniques aren't especially good, but I don't care. We know how to improve them. By making a group take the Implicit Association Test, applying a technique to them, giving them the test again, and seeing how their score changed, we gain the ability to test bias-fighting techniques. I wouldn't want to do this on one person, because the test only has moderate reliability at the individual level. But a group of a few dozen, all practicing the same technique, would be quite sufficient. If another group learns a different technique, we can compare their IAT score improvement and see which technique is better, or if different techniques are better in different circumstances.

Again, there's no reason why this method should be limited to racial biases. No matter how hard I try to evaluate policies on their merits rather than their politics, I am biased towards the US Democratic Party and I know it. This ought to be visible on an IAT, and there ought to be techniques to cure it. I don't know what they are, but I'd like to find them and start testing them.

What about the second method of overcoming bias, routing around it? The IAT is less directly valuable here, but it's not without a role.

In one of the IAT experiments, subjects evaluated essays written by black or white students. This is a fiendishly difficult task upon which to avoid bias. A sneaky researcher can deliberately select essays graded as superior by a blind observer and designate them "white essays", so anyone trying to take the easy way out by giving all essays the same grade can be caught immediately. I like this essay task. It's utterly open to any technique you want to use to reduce bias.

So give someone IATs until you find a group they're especially biased against - black people, Palestinians, Korean-Americans, frequentists; any will do. Then make them grade essays by the control group and the disliked group. Collect statistics correlating IAT bias with essay grading bias. If a person using a special technique to route around mental bias can grade essays more accurately than other people with the same level of IAT bias, that person has routed around their bias successfully.

So: How do we tell if a technique for routing around bias works? Test whether people are better able to conduct a rating task than their IAT scores would predict. How do we test a technique for fighting bias directly? See if it lowers IAT scores. All terribly inconvenient because of the IAT's low effect size and reliability, but with a large enough sample size or enough test-retest cycles the thing could be done. And the psychologists who transformed the Bona Fide Pipeline into the IAT may yet transform the IAT into something even more powerful.

This, then, is one solution to schools proliferating without evidence. With enough research, it could be turned into one of the missing techniques of rationality verification.

 

Footnotes

1: Remember, the IAT is only moderately good at evaluating individuals, and has a bad habit of changing its mind each time someone takes it. Much of what is in this essay would work poorly (though probably still better than nothing) with a simple IAT. But having someone take the IAT ten times over ten days and averaging the results might give a more accurate picture (I don't know of any studies on this). And in any case the IAT is quite good at comparing groups of people with sample size >1. And I expect that souped-up versions of the IAT will be out within a few years; these tests have gotten better and better as time goes on.

Comment author: MBlume 26 March 2009 06:31:47AM 3 points [-]

Quick Poll: How many rationalists meditate? It seems like the mental discipline involved could be highly useful.

For those who do: what sort of training did you use? Did you teach yourself, or find a teacher? What benefits do you perceive from the practice?

Comment author: Yvain 26 March 2009 10:40:22PM 1 point [-]

Another "occasionally but not systematically".

Comment author: Eliezer_Yudkowsky 26 March 2009 11:46:54AM 21 points [-]

So I've got to ask... do my posts not get voted up as much as the other regular posters' because an upvote doesn't seem to signal much, or because people actually don't like my posts that much? Vote up if the former explanation, down if the latter.

Comment author: Yvain 26 March 2009 11:57:12AM *  6 points [-]

I often catch myself using "other Eliezer posts" as the reference class for an Eliezer post, versus "posts in general" as the reference class for everyone else's posts. That holds you to a much higher standard, especially since I best remember your early Overcoming Bias posts where you were picking off low-hanging fruit. It's unfair to you and I'm trying to stop it. Anti-kibbitzer doesn't work here because I go to new posts from the Recent Posts sidebar, plus your writing style's hard to miss.

I guess that counts as an upvote.

In response to Two Blegs
Comment author: Yvain 26 March 2009 09:54:38AM *  4 points [-]

These aren't real blegs! They don't contain a nugget of vanadium ore!
...ahem. Sorry. I agree with both these points. I might try a post on the second one soon.

Comment author: infinite_asshole 25 March 2009 11:33:10PM 15 points [-]

This is an interesting post, but...

Hating Hitler doesn't mean you're biased against Hitler. Likewise, having a belief about a particular ethnic group doesn't mean you're biased for or against them.

Then how do you know what score you should get on the IAT? I don't know what an unbiased score would be, but an equal-for-both-groups score is most likely biased.

In the Israel vs. Palestine case, your answer would depend more on some meta-level decisions than on ironing out another decimal point of bias. For instance: Should a settlement give equal benefits to both sides; should it compensate for historic injustices; should it maximize expected value for the participants; should it maximize expected value for the world?

If you want to maximize expected value for the world, you would end up calculating something like this:

  • Israelis have given us a hugely disproportionate number of the world's famous scientists, musicians, artists, writers, producers, bankers, doctors, and lawyers.
  • Palestinians have given us a hugely disproportionate number of the world's famous suicide bombers.

The answer you would then arrive at will be more different from the answer you would arrive at if "equitable outcome" is your goal, than the difference made by any bias. (Unless you really, really hate lawyers.)

So I don't think this approach gets at any of the hard problems.

Comment author: Yvain 26 March 2009 12:22:39AM *  13 points [-]

Aargh. That's a good point and I clearly need to think about this more. I don't have a clear theory yet, but I'm going to brain-dump my thoughts on this topic.

Then how do you know what score you should get on the IAT? I don't know what an unbiased score would be, but an equal-for-both-groups score is most likely biased.

The score you should get on the IAT should be correlated to your conscious opinion. If you consciously think Palestinians are inferior, then you should be happy with an IAT score showing you think Palestinians are inferior. If you consciously think Palestinians are equal to Israelis, you should be trying to get an IAT score reflecting that equality. It's all about trying to get the unconscious mind to correspond to your rational beliefs.

I consciously assent to the proposition "Palestinians are more likely to be suicide bombers than some other groups, but it's still only a tiny fraction of their population" My unconscious probably believes something closer to "Palestinians = suicide bombers!" Further, my conscious mind stops way short of the proposition "All Palestinians are bad people." My unconscious mind probably believes this second proposition. I'd like my unconscious mind to get way closer to my conscious beliefs in both areas.

My conscious, rational brain believes that most Palestinians are probably decent people who have been driven to extremes by their situation. My conscious mind also believes that the best Middle East peace plan is one where everyone, Israeli or Palestinian, is considered equally deserving of happiness simply because they are human. That's my moral system, and yours may differ. The point is, that is my moral system, and I would like to be able to operate on it. If I'm going around subconsciously thinking that Palestinians are bad and don't deserve happiness, I can't enact my goals.

This ties into the halo effect and the horns effect, where people tend to classify others as either all good or all bad. My belief that Palestinians are sometimes suicide bombers probably makes me think that they're uglier, stupider, and meaner than they really are. A lot of that is mediated through that concept "bad", which is one reason I'm so interested in getting my link between "Palestinians" and "bad" down.

If you wanted to adjust moral value for the Israelis' greater economic value, you'd still need a way to make sure you're not over-adjusting, ie that your unconscious mind doesn't dislike the Palestinians even more than your conscious mind does. Alternately, you'd want to make sure you weren't going soft and that your unconscious mind liked the Palestinians more than your conscious mind thought they deserved. I can't think of an easy way to do that with the IAT, but I bet there's a complicated one if you imagine a hypothetical IAT with perfect reliability and let me get away with a mere proof-of-concept.

Hope that makes sense.

Comment author: pjeby 25 March 2009 10:08:30PM 4 points [-]

You're moving pretty quickly, though, and I have trouble following you at some areas. Maybe in the future break large essays like this into a few blog posts, one for each sub-point.

Heh. This is a way scaled-back version of my original planned first post, which was to jump straight from motivated reasoning into either the Speculator/Savant divide, or the Towards/Away distinctions. I went, "crap, this is getting too long" and pulled the plug before I got to anything really "interesting", figuring that this one at least laid a little bit of groundwork and some references to build a foundation for the rest.

I'm accustomed to being able to get further in one sitting, but that's because my usual writing isn't peppered with references to experimental results and tediously building my case point by point; usually I just rely on metaphor and people's personal experiences as evidence. Here, though, I've noticed that people prefer authorities in the form of citations, to looking at their own personal experiences... so it seems to take a hell of a lot longer to build up statements of any substance.

Which is not to say it's not worth it... discussions on LW, and the preparation for this post, have helped me immensely clarify and simplify certain aspects of my knowledge and work, in ways that will help me teach my self-improvement audience better, not just communicate better on LW.

Comment author: Yvain 25 March 2009 11:26:05PM *  0 points [-]

I'm glad you're paying attention to experimental results. I wouldn't believe you if you didn't :)

Now that I've read it over a few more times, I'm still not sure if I understand correctly. Tell me if this is the right track: the brain tags thoughts or concepts as good or bad by associating them with certain micro-expressions which are the physiological correlates of emotions. When you're reasoning, you are unconsciously trying to generate pleasant emotions by using only those lines of thinking that lead to the micro-expressions associated with pleasant feelings. Generating these pleasant feelings is, on a preconscious level, the desire motivating reasoning.

Also, are you taking as a premise something like the James-Lange theory of emotions? What about something like Reich's theory of muscular armor? (see about a quarter of the way down this page)

In response to The Good Bayesian
Comment author: Kaj_Sotala 25 March 2009 10:30:17PM *  1 point [-]

Not sure of where to ask this question, so I might as well ask it here. (Thought of some old entries that may fit better, but I don't know if comments posted there will get noticed.)

I'm writing an essay on why all religion is objectively false. It is basically a summary of some of Eliezer's main themes and reformulates in my words points found in posts like Outside the Laboratory, Burdensome Details, Dark Side Epistemology, and so on. As such, I feel a bit reluctant to post it in its full length here - I feel like I'd be preaching to the choir, and not contributing anything new. On the other hand, it may be useful to newcomers, and getting a certain lesson reformulated in some other fashion is always useful for learning.

So what I'm basically asking is - when it's done, should I post it in LW in its entirety, or should I just post a link? I realize that it doesn't really make that big of a difference which way I do, but for some reason I still felt like I should ask for the general opinion...

Comment author: Yvain 25 March 2009 10:32:31PM 1 point [-]

On a related note, can we have a LW Open Thread, or some other way to post links or single-sentence points that don't deserve their own topics?

Fight Biases, or Route Around Them?

25 Post author: Yvain 25 March 2009 10:23PM

Continuation of: The Implicit Association Test
Response to: 3 Levels of Rationality Verification

I've not yet seen it pointed out before that we use "bias" to mean two different things.

Sometimes we use "bias" to mean a hard-coded cognitive process that results in faulty beliefs. Take as examples the in-group bias, the recall bias, the bad guy bias, and various other things discovered by Tversky and Kahneman.

Other times, we use "bias" to mean a specific faulty belief generated by such a process, especially one that itself results in other faulty beliefs. For example, Jews are sometimes accused of having a pro-Israel bias. By this we mean that they have a higher opinion of Israel than the evidence justifies; this is a specific belief created by the in-group bias. This belief may itself generate other faulty beliefs; for example, they may have a more negative opinion of Palestinians than the evidence justifies. It is both the effect of a bias, and the cause of other biases.

Let's be clear about this "more than the evidence justifies" bit. Hating Hitler doesn't mean you're biased against Hitler. Likewise, having a belief about a particular ethnic group doesn't mean you're biased for or against them. My Asian friends hate it when people sheepishly admit in a guilty whisper that they've heard Asians are good at academics. Asians are good at academics. Just say "55% chance an average Asian has a GPA above the American population mean" and leave it at that. This is one of Tetlock's critiques of the Implicit Association Test, and it's a good one. I'd probably link Asians to high achievement on an IAT, but it wouldn't be a bias or anything to get upset about.

And let's also be clear about this faulty belief thing. You don't have to believe something for it to be a belief; consider again the skeptic who flees the haunted house. She claims she doesn't belief in ghosts, and she's telling the truth one hundred percent. She's still going to be influenced by her belief in ghosts. She's not secretly supernaturalist any more than someone who gets "strongly biased" on the IAT is secretly racist. But she needs to know she's still going to run screaming from haunted houses, and IAT-takers should be aware they're still probably going to discriminate against black people in some tiny imperceptible way.

Okay, back to the example. So the President appoints Isaac, a synagogue-going Jew, as the new Middle East peace envoy. Due to some amazing breakthrough in the region, both the Israelis and Palestinians agree to accept whatever plan Isaac develops. Isaac's only job is to decide what long-term plan is best for both sides. And he's a good man: he has an honest desire to choose the maximum-utility solution.

Isaac legitimately worries that he has a bias for the Israelis and against the Palestinians. How can he test the hypothesis? He can take a hypothetical souped-up version of the Implicit Association Test1. He finds that yes, he has a strong pro-Israel anti-Palestine bias. Now what does he do?

He can try to route around the bias. This is the approach implicitly endorsed by Overcoming Bias and by rationalism in general. He can take the Outside View and look at successful approaches in other world conflicts. He can use some objective metric to calculate the utility of everything in Israel, and check to make sure neither group is getting an amount disproportionate to their numbers. He can open a prediction market on metrics of success, and implement whatever policies trades at the highest value. All of these will probably improve Isaac's solution a fair bit. But none of them are perfect. In the end, Isaac's the one who has to make a decision that will be underdetermined by all these clever methods, and Isaac is still biased against the Palestinians.

Or he can try to fight the bias.

Diversity workshops try to fight biases directly . These don't work, and that's no surprise. Diversity workshops are telling you, on a conscious level, that minorities really are great people, aren't they? Well, yes. On a conscious level, you already believe that. Isaac already knows, on a conscious level, that the Palestinians deserve a fair solution that protects their interests just as much as the Israelis do. A diversity workshop would be a flashy video in which a condescending narrator explains that point again and again.

We don't have a lot of literature on what does work here, but I predict a few things would help. Make some Palestinian friends, to build mental connections between Palestinians and positive feelings. Learn to distinguish between Palestinian faces. Read works of fiction with sympathetic Palestinian characters. I would say "live in Palestine" but by all accounts Palestine is a pretty grim place; he might do better to live in a Palestinian community in America for a while.

Those techniques aren't especially good, but I don't care. We know how to improve them. By making a group take the Implicit Association Test, applying a technique to them, giving them the test again, and seeing how their score changed, we gain the ability to test bias-fighting techniques. I wouldn't want to do this on one person, because the test only has moderate reliability at the individual level. But a group of a few dozen, all practicing the same technique, would be quite sufficient. If another group learns a different technique, we can compare their IAT score improvement and see which technique is better, or if different techniques are better in different circumstances.

Again, there's no reason why this method should be limited to racial biases. No matter how hard I try to evaluate policies on their merits rather than their politics, I am biased towards the US Democratic Party and I know it. This ought to be visible on an IAT, and there ought to be techniques to cure it. I don't know what they are, but I'd like to find them and start testing them.

What about the second method of overcoming bias, routing around it? The IAT is less directly valuable here, but it's not without a role.

In one of the IAT experiments, subjects evaluated essays written by black or white students. This is a fiendishly difficult task upon which to avoid bias. A sneaky researcher can deliberately select essays graded as superior by a blind observer and designate them "white essays", so anyone trying to take the easy way out by giving all essays the same grade can be caught immediately. I like this essay task. It's utterly open to any technique you want to use to reduce bias.

So give someone IATs until you find a group they're especially biased against - black people, Palestinians, Korean-Americans, frequentists; any will do. Then make them grade essays by the control group and the disliked group. Collect statistics correlating IAT bias with essay grading bias. If a person using a special technique to route around mental bias can grade essays more accurately than other people with the same level of IAT bias, that person has routed around their bias successfully.

So: How do we tell if a technique for routing around bias works? Test whether people are better able to conduct a rating task than their IAT scores would predict. How do we test a technique for fighting bias directly? See if it lowers IAT scores. All terribly inconvenient because of the IAT's low effect size and reliability, but with a large enough sample size or enough test-retest cycles the thing could be done. And the psychologists who transformed the Bona Fide Pipeline into the IAT may yet transform the IAT into something even more powerful.

This, then, is one solution to schools proliferating without evidence. With enough research, it could be turned into one of the missing techniques of rationality verification.

 

Footnotes

1: Remember, the IAT is only moderately good at evaluating individuals, and has a bad habit of changing its mind each time someone takes it. Much of what is in this essay would work poorly (though probably still better than nothing) with a simple IAT. But having someone take the IAT ten times over ten days and averaging the results might give a more accurate picture (I don't know of any studies on this). And in any case the IAT is quite good at comparing groups of people with sample size >1. And I expect that souped-up versions of the IAT will be out within a few years; these tests have gotten better and better as time goes on.

In response to comment by Yvain on The Sacred Mundane
Comment author: Eliezer_Yudkowsky 25 March 2009 09:28:52PM 24 points [-]

Real religious experiences, the sort where you get one, say "Oh, I just saw God" and spend the rest of your life in a monastery trying in vain to capture that sense of connection again

I know an atheist who gets these. She used to think it was future superintelligences talking to her, but eventually she asked herself some very hard questions and managed to realize it was just a brain storm. It's one of the most heroic acts of rationality I've ever seen anyone perform.

But considering that some atheists do get these involuntarily and the vast supermajority of religious folk never get them at all, why call them "religious experiences"?

Comment author: Yvain 25 March 2009 10:02:41PM *  25 points [-]

But considering that some atheists do get these involuntarily and the vast supermajority of religious folk never get them at all, why call them "religious experiences"?

The explanation for this is in the same book from which I took the dhyana quote. I may write a post on it one day, although I worry that an explanation of mysticism by a possibly insane self-confessed magician is a little off-topic for this site.

The short version is that a dhyana experience is completely unconditioned, and the brain quickly sets about conditioning it with cultural experience. Anything that vast and that holy is assumed to be the most powerful entity in the culture of the person who experiences it, usually God. There's also some evidence that the dhyana experience can itself be conditioned by culture, in the same way that a paranoid suffering delusions of persecution for completely biological reasons may interpret it as demons in medieval Europe or the CIA in modern America. Just like the brain throws the label "the CIA" on what ought to be a general persecuted feeling, it throws the label "God", "Jesus", "Allah", "Buddha-nature", "Brahman", "future superintelligence", or whatever else onto what ought to be a general feeling of intense power. This isn't interpreted as a post-hoc attribution; just as the paranoid feels like it's the CIA after them, the Christian feels like they just saw Jesus.

That's what I meant by saying its association with religion was historical and contingent rather than ontological.

Comment author: Yvain 25 March 2009 08:15:10PM 9 points [-]

Very interesting post. If you can do even a fraction of what you say you will, it'll be a spectacular contribution. I already have your blog on my list of things I need to get around to reading, and it just moved up a few places on that list.

You're moving pretty quickly, though, and I have trouble following you at some areas. Maybe in the future break large essays like this into a few blog posts, one for each sub-point.

Comment author: Roko 25 March 2009 01:40:22PM 0 points [-]

That was incredible. I took the gender/careers IAT, and got

Your data suggest a strong association of Male with Career and Female with Family compared to Female with Career and Male with Family.

Whereas I am consciously quite strongly in support of gender equality. It was frightening...

Comment author: Yvain 25 March 2009 06:04:22PM 4 points [-]

Yeah, I agree with Annoyance. Don't worry about it. It's like nurses - you may not have a belief at any level that nurses should be female, but you'd have to be blind not to notice that they usually are. So "nurse" is closer to "female" than "male" in your thing space.

Comment author: Nick_Tarleton 25 March 2009 04:04:12AM *  38 points [-]

If it's easier to place him with the scientists, or there's no difference, that's some evidence we haven't become a cult yet. If it's easier to place him with the cult leaders, we should start worrying.

Really? Don't you think Scientologists would consider L. Ron Hubbard more similar to Einstein than to Jim Jones? I have no data, but I don't imagine cultists develop positive attitudes toward leaders of cults unrelated to their own.

Comment author: Yvain 25 March 2009 06:02:34PM 7 points [-]

Ohhh....good point.

Comment author: aausch 25 March 2009 05:22:17PM 4 points [-]

I tried the online version of the tests, and the priming seemed to me to be built in...

After 40 or 50 training clicks, priming my brain to link "left hand" with "good" and "tested quality", of course it'll take me longer/I'll make more mistakes/etc... when you start juggling the associations around.

Comment author: Yvain 25 March 2009 06:01:18PM *  2 points [-]

Good catch. There is a small order effect on these tests. From the Project Implicit FAQ:

Q: Could the result be a function of the order in which I did the two parts? I had to group one category together with pleasant words first. I then found it difficult when I later had to group the other category with pleasant words.

A: Answer: The order in which tests are administered does make a difference to the overall result in some tests. However, the difference is small and recent changes to the test have sharply reduced the influence of order. Because of this order effect, the orders used for IATs presented on this website are assigned at random. For any data we present, we are careful to be sure that half the test-takers got the A then B order and the other half got the B then A order. With the revised task design, the order has only a minimal influence on task performance. If you want to check whether the order made a difference for you, you can take the test again and complete it if you get assigned to the reverse order. If you do take the test twice in different orders and get different outcomes, the best estimate of your result is intermediate between the two. For more information about the order effect, see this paper

In response to comment by Yvain on The Sacred Mundane
Comment author: Annoyance 25 March 2009 01:35:54PM 4 points [-]

"I took hashish once and started seriously questioning the nature of mind and experience."

That's wonderful... but is there any particular reason why you couldn't have done the same with a cup of coffee?

Was it something special about the hashish experience, or merely that it was so novel that it caused you to pay a lot of attention to it? What if you paid that much attention to the things you consider mundane and banal?

Comment author: Yvain 25 March 2009 05:29:56PM *  26 points [-]

There's a risk here of using "mundane experience" as an applause light.

Consider the equivalent query - doctors have learned a lot about the brain by studying stroke victims. For example, one reason we know that the frontal cortex is responsible for inhibition is because people who get frontal cortex injuries lose their inhibition.

You can go up to a neurologist and say "That's wonderful...but couldn't you have learned the same thing if you really closely observed the brain of a normal person?" But why should the neurologist deny himself a useful tool just because it's not mundane enough?

You can learn arbitrarily much by contemplating everyday life. Eliezer theorizes that a superintelligence could deduce General Relativity just by watching an apple fall. But that doesn't mean you should turn your nose up at Einstein for using the perihelion of Mercury. There's no such thing as cheating in rationalism.

In response to The Sacred Mundane
Comment author: Yvain 25 March 2009 12:37:10PM *  36 points [-]

There's a difference between "moving experience" and "spiritual experience" that I think both Adam Frank and Eliezer are too quick to dismiss. Seeing a space shuttle blast off is inspirational, but as Eliezer correctly points out there's nothing private or especially religious about it.

Real religious experiences, the sort where you get one, say "Oh, I just saw God" and spend the rest of your life in a monastery trying in vain to capture that sense of connection again, are much more likely to be some very exotic neurological event. Consider for example the commonly remarked upon similarity of "trips" on entheogenic drugs, which we know are screwing with neurotransmission in some way, to mystical experiences.

This sort of a spiritual experience really is absolutely private and absolutely incommunicable. Those who have felt it describe it as a feeling completely alien to and much more powerful than any other feeling they've ever had - which seems completely plausible to me if it's really some sort of weird realignment of cognitive processes. How are you supposed to share or communicate a high-level reprogramming of your brain to someone else? How is a non-neurologist supposed to describe it in any terms other than what they've "experienced"?

This is a passage on Dhyana (a Sanskrit word transliterated into Japanese as "Zen", indicating an extremely high state of mystical achievement) by a certain famous yogi:

In discussing Dhyana, then, let it be clearly understood that something unexpected is about to be described. We shall consider its nature and estimate its value in a perfectly unbiassed way, without allowing ourselves the usual rhapsodies, or deducing any theory of the universe. One extra fact may destroy some existing theory; that is common enough. But no single fact is sufficient to construct one.

In the course of our concentration we noticed that the contents of the mind at any moment consisted of two things, and no more: the Object, variable, and the Subject, invariable, or apparently so. By success in Dharana the object has been made as invariable as the subject. Now the result of this is that the two become one. This phenomenon usually comes as a tremendous shock. It is indescribable even by the masters of language; and it is therefore not surprising that semi-educated stutterers wallow in oceans of gush.

All the poetic faculties and all the emotional faculties are thrown into a sort of ecstasy by an occurrence which overthrows the mind, and makes the rest of life seem absolutely worthless in comparison.

Good literature is principally a matter of clear observation and good judgment expressed in the simplest way. For this reason none of the great events of history (such as earthquakes and battles) have been well described by eye-witnesses, unless those eye-witnesses were out of danger. But even when one has become accustomed to Dhyana by constant repetition, no words seem adequate.

I doubt Adam Frank has ever had one of these experiences, but some of the people he reads have, and some of the people whom the people he reads read have, and he's taken them and misinterpreted them as equivalent to going to Newgrange and being inspired by it. I went to Newgrange once and thought it was pretty neat. I took hashish once and started seriously questioning the nature of mind and experience.

[note: I am not claiming that normal go-to-church-each-week religion is particularly related to this sort of "religious experience". That both of them are grouped together is more of a historical fact than an ontological one.]

The Implicit Association Test

24 Post author: Yvain 25 March 2009 12:11AM

Continuation of: Bogus Pipeline, Bona Fide Pipeline
Related to: The Cluster Structure of Thingspace

If you've never taken the Implicit Association Test before, try it now.

Any will do. The one on race is the "classic", but the one on gender and careers is a bit easier to watch "in action", since the effect is so clear.

The overwhelming feeling I get when taking an Implicit Association Test is that of feeling my cognitive algorithms at work. All this time talking about thingspace and bias and categorization, and all of a sudden I have this feeling to attach the words to...

...which could be completely self-delusional. What is the evidence? Does the Implicit Association Test work?

Let the defense speak first1. The Implicit Association Test correctly picks up control associations. An IAT about attitudes towards insects and flowers found generally positive attitudes to the flowers and generally negative attitudes to the insects (p = .001), just as anyone with their head screwed on properly would expect. People's self-reports were also positively correlated with their IAT results (ie, someone who reported loving flowers and hating insects more than average also had a stronger than average IAT) although these correlations did not meet the 95% significance criterion. The study was repeated with a different subject (musical instruments vs. weapons) and similar results were obtained.

In the next study, the experimenters recruited Japanese-Americans and Korean-Americans. Japan has been threatening, invading, or oppressing  Korea for large chunks of the past five hundred years, and there's no love lost between the two countries. This time, the Japanese-Americans were able to quickly match Japanese names to "good" stimuli and Korean names to "bad" stimuli, but took much longer to perform the opposite matching. The Korean-Americans had precisely the opposite problem, p < .0001.  People's self-reports were also positively correlated with their IAT results (ie, a Korean who expressed especially negative feelings towards the Japanese on average also had a stronger than average IAT result) to a significant level.

There's been some evidence that the IAT is pretty robust. Most trivial matters like position of items don't much much of a difference. People who were asked to convincingly fake an IAT effect couldn't do it. If the same person takes the test twice, there's a correlation ofabout .6 between the  two attempts2. There's a correlation of .55 between the Bona Fide Pipeline and the IAT (the IAT wins all competitions between the two; it produces twice as big an effect size). There's about a .24 correlation between explicit attitude and IAT score, which is significant at the 90% but not the 95% level; removing certain tests where people seem especially likely to lie on their explicit attitude takes it up to 95. When the two conflict, the IAT occasionally wins. In one study, subjects were asked to evaluate male and female applicants for a job. Their observed bias against women correlated more strongly with their scores on a gender bias IAT than with their own self-report (in other experiments in the same study, explicit self-report was a better predictor. The experimenters concluded both methods were valuable in different areas)

Now comes the prosecution. A common critique of the test is that the same individual often gets two completely different scores taking the same test twice. As far as re-test reliability goes, .6 correlation is pretty good from a theoretical point of view, but more than enough to be frequently embarrassing. It must be admitted: this test, while giving consistent results for populations, is of less use for individuals wondering how much bias they personally have.

Carl Shulman would be heartbroken if I didn't mention Philip Tetlock, so here goes. This is from Would Jesse Jackson Fail the Implicit Association Test?, by Tetlock and Arkes (2004):

Measures of implicit prejudice are based on associations between race-related stimuli and valenced words. Reaction time (RT) data have been characterized as showing implicit prejudice when White names or faces are associated with positive concepts and African-American names or faces with negative concepts, compared to the reverse pairings. We offer three objections to the inferential leap from the comparative RT of different associations to the attribution of implicit prejudice: (a) The data may reflect shared cultural stereotypes rather than personal animus, (b) the affective negativity attributed to participants may be due to cognitions and emotions that are not necessarily prejudiced, and (c) the patterns of judgment deemed to be indicative of prejudice pass tests deemed to be diagnostic of rational behavior.

In other words, there are a bunch of legitimate reasons people might get negative IAT scores. Any connection whatsoever between black people and negative affect will do. It could be the connection that black people generally have low status in our society. It could be that a person knows of all the prejudices against black people without believing them. It could be that a person has perfectly rational negative feelings about black people because of their higher poverty rate, higher crime rate, and so on. Or it could be somethng as simple as that, for whites, black people are the out-group.

...this actually isn't much of a prosecution at all. I consider myself a moderate believer in the IAT, and I think it all sounds pretty reasonable.

What most IAT detractors I've read want to make exquisitely clear is that you can't hand someone an IAT, find an anti-black bias, and say "Aha! He's a racist! Shame on him!"3

I think this is pretty obvious4. You can hold beliefs on more than one level. A person may believe there is a dragon in his garage, yet not expect an experiment to detect it. A skeptic may disbelieve in ghosts, but be afraid of haunted houses. A stroke victim may deny an arm is hers while admitting it is attached to her body. And it's supposed to be news that you can give black people some sort of vague negative connotation on a nonconscious level without being Ku Klux Klan material?

There is a certain segment of society which interprets the sun rising in the morning as evidence of racism. It is not surprising that this segment of society also interprets the IAT as evidence for racism. I myself think racism is a bad word. Not in the way "shit" is a bad word, but in the way "wiggin" is a bad word. It divides experience in a perverse way, drawing a boundary such that Adolf Hitler ends up in the same category as the guy who feels a pang of guilty fear late at night when he sees a big muscular black guy walking towards him5. Taboo the word "racism", "prejudice", and any other anti-applause-light6, and a lot of the IAT debate loses its meaning.

Which is good, because I think the IAT is about much more than who is or isn't racist. The IAT is a tool for measuring distances in thingspace.

Thingspace, remember, is the sort of space in which we draw categories7. "Chair" is a useful category because it describes a cluster of things that are close together in concept-space in a certain way: stools, rocking chairs, office chairs, desk chairs, et cetera. "Furniture" is another useful word because it describes another cluster, one that includes the chair cluster and other concepts nearby. Quok, where a "quok" is defined as either a chair or Vladimir Lenin, is a useless category, because Lenin isn't anywhere near all the other members.

Speaking of communists, remember back when East and West Germany got reunited? And remember a little further back, when North and South Vietnam got reunited too? Those reunifications, no matter how you feel about them politically, were natural links between culturally and historically similar regions. But imagine trying to unite East Germany with South Vietnam, and West Germany with North Vietnam. The resulting countries would be ungovernable and collapse in a matter of weeks.

If you associate white people with good things, and black people with bad things, then forming the categories "white and good" and "black and bad" is like reuniting East and West Germany. You're drawing a natural border around a compact area of the map. But being forced into the categories "white and bad" and "black and good" is about as natural as trying to merge East Germany and South Vietnam into the new country "Southeast Vietnermany". You're drawing an arbitrary boundary around two completely unrelated parts of the map and then begging in vain for the disgruntled inhabitants to cooperate with each other.

If you provoke a war between the reunified Germany and Southeast Vietnermany, and watch which side coordinates its forces better, you get the Implicit Association Test.

Why would we want to measure distance in thingspace? Loads of reasons. Take a set of pictures of famous cult leaders, mix them with a set of pictures of famous scientists, and test Less Wrong readers' reaction times associating a picture of Eliezer Yudkowksy's face with either set8. If it's easier to place him with the scientists, or there's no difference, that's some evidence we haven't become a cult yet. If it's easier to place him with the cult leaders, we should start worrying.

Tomorrow: some more serious applications to rationality.

 

Footnotes:

1: Most of these results taken from this, this, and this study.

2: There's some evidence that priming can change your IAT score. For example, subjects shown a picture of a happy black family enjoying a picnic just before an IAT got lower bias scores than a control group who didn't see the picture. And before condemning the test too much for its tendency to give different scores on different occasions, remember back to your school days when you'd have to take endless quizzes on the same subject. Occasionally just by chance you'd get a spread of ten point or so, and if you were on the borderline between passing and failing, you might very well pass one test and fail another test on the exact same material. This doesn't mean grade school tests don't really measure your knowledge, just that there's always a bit of noise. The IAT noise is greater, but not overwhelmingly so.

3: There's also a fear someone might use it for, say, evaluating applicants for a job. Due to its weakness as an individual measurement and the uncertainty about how well it predicts behavior, this would be a terrible idea.

4: Full disclosure: Despite strongly opposing prejudice on a conscious level and generally getting along well with minorities in my personal life, I get assessed as moderately biased on the racism IAT. I had some memorable bad experiences with certain black people in my formative years, so this doesn't much surprise me.

5: In fact, Jesse Jackson (note for non-Americans: a well-known black minister and politician who speaks out against racism) himself admits to occasionally having these pangs of guilty fear - hence the name of Tetlock's article.

6: I think Eliezer once coined a term for the opposite of "applause light", for things like "racism" and "scientism" invoked only so people can feel good about hating them, but I can't seem to find it. Can someone refresh my memory?

7: I was split on whether to use the term thing-space or concept-space here. Eliezer uses concept-space in a very particular way, but "good" and "black" seem much more concepts than things. I eventually went with thing-space, but I'm not happy about it.

8: This is a facetious example. It's possible in theory, but there would be so much to control for that any result would be practically meaningless.

Comment author: Mike 24 March 2009 10:33:20PM *  3 points [-]

I took many tests like this via Project Implicit online. I'm not sure the results are very reliable for individuals, though that is not to say they are unreliable on average.

For instance, I took a racism test in demonstration mode. I was anxious about the exam as I took it, and in particular I was distracted because I had figured out how it worked and I was self-conscious about making "mistakes" (responding more quickly in some scenarios by fluke) or sort-of secretly biasing the result because I didn't want myself to be racist. In the end it told me I had a moderate automatic preference for whites over blacks. Although I do not consider myself racist, I was willing to accept this because I grew up in the Midwest with many racist peers and over the course of my life have had very little exposure to black people.

But then I encountered the same test as part of the experimental section of the site. This time I was much more relaxed because I had come to terms with the previous results. Yet, after this test, it told me I had no automatic preference for whites or blacks.

I don't know whether my performance during the first test was skewed because I was anxious and distracted, whether my second test was skewed because I had taken the test before, or whether there is some large uncertainty and my "true" score is simply between these. I also don't know if my seeing how the experiment works -- and then being distracted by it -- is common or rare.

Comment author: Yvain 24 March 2009 11:01:07PM 2 points [-]

Mike: Studies find a correlation of .6 between the same individual taking the test more than once, so 40% of the test score is always going to be the vagaries of each trial. They also find that "practice" on the IAT explains 15% of variance or so. More on this in a moment.

In response to Levels of Power
Comment author: Yvain 24 March 2009 05:16:18PM *  11 points [-]

The idea is potentially useful but also potentially a giant minefield. Anything that can be hijacked and turned into a stupid rat race for status will be. I spent most of my time as a martial artist wishing I was a black belt, rather than wishing I was good at martial arts. Which is probably why I never got a black belt.

If we want to do this, far better it apply to lessons rather than people. It should be usable only the way you used it on the top of this post: as a way of stating a difficulty level. This is denotatively the same (if level 1.5 posts are appropriate to me, I must be a level 1.5 rationalist) but connotatively very different. There should be a holy unbreakable taboo upon ever mentioning levels in connection with a person.

But before I actually started putting the numbers on posts, I would want a much more precise description of what they mean. If you listed ten or twenty diverse OB posts along with the level you would assign each, that would help.

Bogus Pipeline, Bona Fide Pipeline

22 Post author: Yvain 24 March 2009 12:10AM

Related to: Never Leave Your Room

Perhaps you are a psychologist, and you wish to do a study on racism. Maybe you want to know whether racists drink more coffee than non-racists. Sounds easy. Find a group of people and ask them how racist they are, then ask them how much coffee they drink.

Problem: everyone in your study says they're completely non-racist and some of their best friends are black and all races are equally part of this vast multicolored tapestry we call humanity. Maybe some of them are stretching the truth here a bit. Until you figure out which ones, you're never going to find out anything interesting about coffee.

So you build a foreboding looking machine out of gleaming steel, covered with wires and blinking lights. You sit your subjects down in front of the machine, connect them to its electrodes, and say as convincingly as possible that it is a lie detector and they must speak the truth. Your subjects look doubtful. Didn't they hear on TV that lie detectors don't really work? They'll stick to their vehement assertions of tolerance until you get a more impressive-looking machine, thank you.

You get smarter. Before your experiment, you make the subjects fill in a survey, which you secretly copy while they're not looking. Then you bring them in front of the gleaming metal lie detector, and dare them to try to thwart it. Every time they give an answer different from the one on the survey, you frown and tell them that the machine has detected their fabrication. When the subject is suitably impressed, you start asking them about racism.

The subjects start grudgingly admitting they have some racist attitudes. You have invented the Bogus Pipeline.

The Bogus Pipeline is quite powerful. Since its invention in the 70s, several different studies demonstrate that its victims will give significantly less self-enhancing answers to a wide variety of questions than will subjects not connected to the machinery. In cases where facts can be checked, Pipeline subjects' answers tend to be more factually correct than normal subjects'.

In one of the more interesting Bogus Pipeline experiments, Millham and Kellogg wanted to know how much of a person's average self-enhancement is due to self-deception biases, and how much is due to simple lying. They asked people some questions about themselves under normal and Pipeline conditions, using the Marlowe-Crowne scale. This scale really deserves a post of its own, but the short version is that it asks you some loaded questions, and if you take them as an opportunity to say nice things about yourself, you get marked down as a self-enhancer. There was a correlation of .68 between Marlowe-Crowne scores in normal and Pipeline conditions. If we accept that no one deliberately lies under the Pipeline, that means we now know how much self-enhancement is, on average, self-deception rather than deliberate falsehood (tendency towards deliberate falsehoods correlated .37 with Marlowe-Crowne.1)

Interesting stuff. But you still don't know whether racists drink more coffee! Your Bogus Pipeline only eliminates part of the self-enhancement in your subjects' answers. If you want to solve the coffee question once and for all, you can't count on a fake mind-reading device. You need a real mind-reading device. And in the mid 90s, psychology finally developed one.

The Bona Fide Pipeline is far less impressive-looking than the Bogus Pipeline. Though the Bogus Pipeline tries as hard as it can to scream "mind-reading device", the Bona Fide Pipeline has a vested interest in preventing its victims from realizing their minds are being read. It is a simple computer terminal.

The Pipeline uses a complicated process to disguise itself as an ordinary study on distraction or face recognition or somesuch, but the active ingredient is this: the subjects play a game where they must hit one key (perhaps "A") if the screen displays a good word (for example "wonderful"), and a different key (perhaps "L") if the screen displays a bad word (for example "ugly").

But before it gives you the word, it shows you a picture of a white person or a black person. Remember priming? That picture of a black person is going to prime your brain's concept of "black person" and any concepts you associate with "black person". If you have racist attitudes, "bad" is one concept you associate with "black person". You're going to have a very easy time recognizing "ugly" as a bad word, because your "bad" concept is already activated. But you're going to have a harder time recognizing "wonderful" as a good concept, because your brain is already skewed in the opposite direction. It's not impossible, it's just going to take a few hundred more milliseconds. Each of which the Bona Fide Pipeline is recording and processing. At the end, it spits out a score telling you that you took an average of three hundred milliseconds longer to recognize good words when primed with black people's pictures than white people's pictures.

Does this actually work? The original study (Fazio et al, 1995) tested both whites and blacks, and found the whites were more likely to be prejudiced against blacks than the blacks were, which makes sense. In the same study, a black experimenter conversed with the subjects for a while, and rated the quality of the interaction by a typically rigorous rubric. This fuzzy unscientific measure of racist behavior correlated well with the Pipeline's data for the individuals involved. A study by Jackson (1997) find that people who score high on prejudice by Pipeline measures on average give lower scores to an essay written by a student known to be black.

The Bona Fide Pipeline has lately been superseded by its younger, sexier, Harvard-educated cousin, the IAT. More on that, the associated controversy, and the relevance to rationality tomorrow.

Footnotes:

1: I doubt that deceptions can be separated cleanly into self-deception and deliberate falsehood like this. More likely there are many different shades of grey, and the Bogus Pipeline captures some but not all of them.

Comment author: Yvain 23 March 2009 04:13:02PM *  7 points [-]

From a poster's perspective: it is very hard to tell which ideas your audience considers beginner-level and which they consider advanced-level. Especially when the audience is as diverse and self-selected as at LW. I've posted a few times asking "Hey, does everyone here already know X or not?" and I've rarely gotten the answer I expected.

Responses to my post last night ranged from "this is obvious" to "this is wrong" to "this acronym could be useful" to "this was one of my favorite posts yet". I don't quite know what to do with that. Right now I am erring on the side of caution; I'd rather write something obvious to everyone than skip an inferential distance somewhere.

Upvoting ought to be the main feedback mechanism here, but right now I worry that a well-written true (but obvious) article will get voted up just because it's well-written and true, and everyone figures it will probably help someone else. Maybe make a rule that you should not upvote a post unless it teaches you something? Or maybe end a post whose difficulty level you're not sure of with "Please rate this as too obvious, okay, or too hard"?

EDIT: It's also hard to remember if something has already been covered on Overcoming Bias (see: source confusion). There's not any nice list of Robin or the other writers' posts like there is of Eliezer's, is there?

Comment author: PhilGoetz 23 March 2009 03:33:25AM 15 points [-]

These are funny. But some are from a website about Chuck Norris! Don't incite Chuck's wrath against Eliezer.

If Chuck Norris and Eliezer ever got into a fight in just one world, it would destroy all possible worlds. Fortunately there are no possible worlds in which Eliezer lets this happen.

Comment author: Yvain 23 March 2009 10:43:10AM 51 points [-]

All problems can be solved with Bayesian logic and expected utility. "Bayesian logic" and "expected utility" are the names of Eliezer Yudkowsky's fists.

Comment author: SoullessAutomaton 23 March 2009 02:30:29AM 46 points [-]

Eliezer Yudkowsky can make AIs Friendly by glaring at them.

And the first action of any Friendly AI will be to create a nonprofit institute to develop a rigorous theory of Eliezer Yudkowsky. Unfortunately, it will turn out to be an intractable problem.

Comment author: Yvain 23 March 2009 10:42:15AM 49 points [-]

Transhuman AIs theorize that if they could create Eliezer Yudkowsky, it would lead to an "intelligence explosion".

Comment author: Demosthenes 22 March 2009 11:15:04PM *  2 points [-]

Speaking of themes; I guess one thing that bothered me about this post (many of your other posts are very good) is that this post doesn't seem to serve a point; questioning assumptions is often brought up on OB and LW and asking others to be more precise in describing their assumptions is also very common. Any connection to positivism here seems very tenuous; criticizing positivism has little or no impact on soft-positivism.

I feel that there are many other OB and LW posts that would address this issue more effectively and it might be better to just make this page an index of them as opposed to a full post.

That said, there is no way to easily recognize a lot of the themes here and LW in particular runs the risk of just becoming a repository of the same things over and over again.

Are we really refining the art of human rationality here?

Comment author: Yvain 22 March 2009 11:28:32PM 2 points [-]
Comment author: ciphergoth 22 March 2009 11:10:56PM 4 points [-]

For some reason I'm a critic this evening, forgive me, but I don't think this one meets the standards of your previous articles; it feels to me that this doesn't move us far beyond the widely appreciated fact that for every portion of direct implication an utterance has, there's usually a generous side-salad of connotation. Am I missing a key novel move?

Comment author: Yvain 22 March 2009 11:24:56PM *  12 points [-]

No. I almost put a footnote in saying something like "But you already knew all this, didn't you?"

The reason I wrote it anyway is because of the last four paragraphs. The particular debate failure mode I outlined there - Person A makes a denotatively true but connotatively controversial statement, Person B is offended by this statement so tries to argue against it poorly or make some similarly offensive statement - is one of the most common problems with the arguments I hear. I want to be able to say "Hey, guys, adboc" and give a link to this article.

It also fills in something I thought was a bit of a hole in last night's post. A statement like "Islam is a religion of peace" is a "dangling variable" underdetermined by the empirical facts it purports to relate to. So why does it matter which way we evaluate the proposition? It matters because of all the associated connotations. Since it has practically no denotation, saying "Islam is a religion of peace" is just saying "Please accept all these connotations here" without even the saving grace of a denotation to attach them to.

...tell me what you thought of that preceding paragraph. I can't decide whether it's obvious and a rehash, whether it's uninteresting, or whether it deserves a post of its own.

Comment author: badger 22 March 2009 11:02:49PM 0 points [-]

Thanks for the correction. This makes much more sense as a technique than a general principle.

Comment author: Yvain 22 March 2009 11:05:36PM 1 point [-]

Added your acronym to main post. Please do not doci. Adboc instead.

Comment author: Anatoly_Vorobey 22 March 2009 10:14:43PM *  16 points [-]

This may be excessive nit-picking, I'm not sure; but I just don't think that your examples of the Connotation Game provide connotations as you define them. "demagogue" and "eloquent" have different denotations - it's not just the emotional aura of the words, their actual meanings are different. Ditto for "patriotic" and "jingoistic" and so forth. The point of the game is that the same person/act can be described by flattering, neutral and disparaging words, but even though the same behavior is described by different words, the meanings of the words are different. The game points out inherent subjectivity present in those meanings.

If you used e.g. "black", "colored" and "African-American", then it'd be about connotations. But that's not how the game is played.

More generally, I think you're trying to give the concept of a connotation a larger role than it's used to. Connotations are about differences in phrases like "ultra-rich", "stinking rich", and "independently wealthy". In the phrase about the ultra-rich you quote, it seems difficult to assign the subtext being carried to connotations, unless you want to use "connotation" in the sense of "subtext, any implied meaning". But I don't think those shoes will fit the word.

Subtexts are very important, and are used insidiously all the time. They're not always emotional; sometimes they exploit logical structure, as in the canonical example "Have you stopped beating your wife?" It's difficult to stand your ground against them, and forcing them into the open, demanding that they be made explicit, as you suggest, is a useful technique. Unfortunately, it usually won't work on someone who feels entitles to use a subtext against you - due to either intentional demagoguery or misguided self-righteousness, for instance.

One framework that seems to make a careful study of implied meanings is that of framing. I don't really know what they're doing with this and whether it's more than a useful metaphor. But, to repeat myself, I think that "connotations" might be too straight a jacket to use for the notion.

Comment author: Yvain 22 March 2009 11:03:29PM 2 points [-]

You could be right; I have no formal training in this field. It sounds like you don't like "subtext". Suggest another word and I'll switch to it.

Comment author: badger 22 March 2009 10:35:03PM *  20 points [-]

This is one of my favorite posts yet, but I'm not sure I understand your full chain of reasoning. I understand you to be arguing that we should only be affected by the denotational content of a statement, and ignore connotations as best we can. I entirely agree we shouldn't confuse the two, but I don't see how to go from that to your full conclusion. Is the danger of confusion so great it is worth giving up the extra expressiveness of connotation? I'd appreciate some clarification.

I really like the idea of an acronym, but I'd like one that can be used naturally as a verb. My best shot is "agree denotationally but oject connotatively", e.g I adboc that the rich party while the poor starve.

Comment author: Yvain 22 March 2009 10:55:12PM *  10 points [-]

I don't think we should eliminate connotations from the language, if that's what you're asking.

But in cases where we're trying to use X-Treme Rationality on things that have otherwise resisted debate, we have to use special techniques make the problem more tractable. And one such technique is positivist thinking and putting a big wall up between connotations and denotations.

I like your acronym.

Comment author: Demosthenes 22 March 2009 10:19:18PM *  5 points [-]

This post just appeared at Language log and has this to say:

Language Log could devote a thousand posts to the project of underlining and elaborating the ways in which grammar does not protect us against misunderstanding the sound of an uttered name, and logic does not protect us against what we say having double meaning. Come to think of it, the thousand posts may already have been written: there are over 5,500 old posts searchable here and already over 1,200 new ones on the present server searchable here.

http://languagelog.ldc.upenn.edu/nll/?p=1258

Asking others to clarify their assumptions is good, but translating them into positivist language might not be as precise as it sounds

Comment author: Yvain 22 March 2009 10:50:18PM *  4 points [-]

One theme of OB and LW has been to take the fuzzy complexity of the real world and show that in principle at some level it's related to something precise. We can't actually do Bayesian math in our heads for real-world calculations, but just knowing how to work the ideal case protects us against certain real-world errors. Likewise, Eliezer's meta-ethics reduces morality down to some horrendously complex thing that we can never calculate, but it's nice to know that morality does reduce to something when we're wondering whether it exists at all, whether it's all relative, or so on.

The real positivists thought they could reduce all language to their positivism and spent thirty years trying. I don't think I'm going to do that in a few days of posting about stuff on a blog. But if I can sketch a few really-large-scale things and then let people's common sense fill in the blanks, that'll still be better than nothing.

Comment author: badger 22 March 2009 09:41:53PM 45 points [-]

Teachers try to guess Eliezer Yudkowsky's password.

Comment author: Yvain 22 March 2009 09:53:13PM *  90 points [-]

Eliezer Yudkowsky's map is more accurate than the territory.

Comment author: Alicorn 22 March 2009 04:39:38PM *  2 points [-]

I used to have the idea that finding flaws in something (a piece of writing or entertainment or an idea or a person) made me better than the person or the creator of the thing I was criticizing. Then I realized two things which got me to stop: 1) Critics are parasites; they don't generally produce anything that valuable and entertaining themselves, and even beautifully written reviews are pretty low on my list of things to read for edification or fun. 2) When I go around finding flaws in everything, I stop enjoying it, and living a life where I can't enjoy anything I read or hear or see is not pleasant.

So now my strategy is to like things and people I'm inclined to like, but remain confident in my so-far-unfailing ability to find fault with them if I decide I need or want to do that. Being accused of slavish devotion to something is one of the things that can make me want to turn critic-mode back on, even though that only winds up proving a lack of slavish devotion after the fact (since I will tend not to notice the sorts of flaws I point out in critic-mode until I actually turn on critic-mode).

All of that having been said, Paul Graham is awesome, so is Eliezer, and GEB is overrated (either that or it was ruined for me when I took a class on it with non-philosophers teaching it; I do have a history of hating anything I'm obliged to read for school).

Comment author: Yvain 22 March 2009 09:51:50PM *  1 point [-]

When I go around finding flaws in everything, I stop enjoying it, and living a life where I can't enjoy anything I read or hear or see is not pleasant.

Having recently posted on the relevance of Pope's poetry to rationalism, I can't help quoting him one more time here:

Avoid extremes, and shun the faults of such
Who still are pleased too little or too much
Those minds, like stomachs, are not always best
That nauseate all, and nothing can digest

In response to Cached Selves
Comment author: Yvain 22 March 2009 09:28:11PM *  6 points [-]

This is a great post, especially all of the technique suggestions.

Socially Required Token Disagreement: I'm especially surprised by the "drive safely" study - and it's especially weird since "keeping America beautiful" would seem to contradict putting a big ugly sign on your front lawn. Maybe the effect wasn't through the person's support for vague feel-good propositions, but through their changed attitude to following requests by strangers knocking on their door.

Comment author: AnnaSalamon 22 March 2009 09:03:23PM *  2 points [-]

The other thing is the Outside View summed up by the proverb "If two people think alike, one of them isn't thinking." In the majority of cases I observe ... that person is a sheeple and that leader has a cult.

Do you have a mechanistic unpacking (even a guess would be helpful) of what it is to be a "sheeple" or a "cult", and of what harms come from being a "sheeple"? Given Aumann, I'm more inclined to say that if two people have different beliefs, at least one of them isn't thinking.

That said, your point about respecting outside views is reasonable. Are you trying to avoid replacing the outside-presumed "badness" of cults/sheeple with understood mechanisms, so as to retain any usefulness that might be in the received heuristics and that you might not understand the mechanisms behind?

Comment author: Yvain 22 March 2009 09:15:45PM 6 points [-]

By sheeple and cult, I mean people whose good judgment is clouded by the mechanisms described in the Affective Death Spiral sequence.

Comment author: Yvain 22 March 2009 09:07:07PM *  97 points [-]

Ooh, this is fun.

Robert Aumann has proven that ideal Bayesians cannot disagree with Eliezer Yudkowsky.
Eliezer Yudkowsky can make AIs Friendly by glaring at them.
Angering Eliezer Yudkowsky is a global existential risk
Eliezer Yudkowsky thought he was wrong one time, but he was mistaken.
Eliezer Yudkowsky predicts Omega's actions with 100% accuracy
An AI programmed to maximize utility will tile the Universe with tiny copies of Eliezer Yudkowksy.

When Truth Isn't Enough

91 Post author: Yvain 22 March 2009 08:23PM

Continuation of: The Power of Positivist Thinking

Consider this statement:

The ultra-rich, who control the majority of our planet's wealth, spend their time at cocktail parties and salons while millions of decent hard-working people starve.

A soft positivist would be quite happy with this proposition. If we define "the ultra-rich" as, say, the richest two percent of people, then a quick look at the economic data shows they do control the majority of our planet's wealth. Checking up on the guest lists for cocktail parties and customer data for salons, we find that these two activities are indeed disproportionately enjoyed by the rich, so that part of the statement also seems true enough. And as anyone who's been to India or Africa knows, millions of decent hard-working people do starve, and there's no particular reason to think this isn't happening at the same time as some of these rich people attend their cocktail parties. The positivist scribbles some quick calculations on the back of a napkin and certifies the statement as TRUE. She hands it the Official Positivist Seal of Approval and moves on to her next task.

But the truth isn't always enough. Whoever's making this statement has a much deeper agenda than a simple observation on the distribution of wealth and preferred recreational activities of the upper class, one that the reduction doesn't capture.


Philosophers like to speak of the denotation and the connotation of a word. Denotations (not to be confused with dennettations, which are much more fun) are simple and reducible. To capture the denotation of "old", we might reduce it to something testable like "over 65". Is Methusaleh old? He's over 65, so yes, he is. End of story.

Connotations0 are whatever's left of a word when you subtract the denotation. Is Methusaleh old? How dare you use that word! He's a "senior citizen!" He's "elderly!" He's "in his golden years." Each of these may share the same denotation as "old", but the connotation is quite different.

There is, oddly enough, a children's game about connotations and denotations1. It goes something like this:

I am intelligent. You are clever. He's an egghead.
I am proud. You are arrogant. He's full of himself.
I have perseverance. You are stubborn. He is pig-headed.
I am patriotic. You're a nationalist. He is jingoistic.

Politicians like this game too. Their version goes:

I care about the poor. You are pro-welfare. He's a bleeding-heart.
I'll protect national security. You'll expand the military. He's a warmonger.
I'll slash red tape. You'll decrease bureaucracy. He'll destroy safeguards.
I am eloquent. You're a good speaker. He's a demagogue.
I support free health care. You support national health care. He supports socialized health care.

All three statements in a sentence have the same denotation, but very different connotations. The Connotation Game would probably be good for after-hours parties at the Rationality Dojo2, playing on and on until all three statements in a trio have mentally collapsed together.

Let's return to our original statement: "The ultra-rich, who control the majority of our planet's wealth, spend their time at cocktail parties and salons while millions of decent hard-working people starve." The denotation is a certain (true) statement about distribution of wealth and social activities of the rich. The connotation is hard to say exactly, but it's something about how the rich are evil and capitalism is unjust.

There is a serious risk here, and that is to start using this statement to build your belief system. Yesterday, I suggested that saying "Islam is a religion of peace" is meaningless but affects you anyway. Place an overly large amount of importance on the "ultra-rich" statement, and it can play backup to any other communist beliefs you hear, even though it's trivially true and everyone from Milton Friedman on down agrees with it. The associated Defense Against The Dark Arts technique is to think like a positivist, so that this statement and its reduced version sound equivalent3.

...which works fine, until you get in an argument. Most capitalists I hear encounter this statement will flounder around a bit. Maybe they'll try to disprove it by saying something very questionable, like "If people in India are starving, then they're just not working hard enough!" or "All rich people deserve their wealth!4 "

Let us take a moment to feel some sympathy for them. The statement sounds like a devastating blow against capitalism, but the capitalists cannot shoot it down because it's technically correct. They are forced to either resort to peddling falsehoods of the type described above, or to sink to the same level with replies like "That sounds like the sort of thing Stalin would say!" - which is, of course, denotatively true.

What would I do in their position? I would stand tall and say "Your statement is technically true, but I disagree with the connotations. If you state them explicitly, I will explain why I think they are wrong."

YSITTBIDWTCIYSTEIWEWITTAW is a little long for an acronym, but ADBOC for "Agree Denotationally But Object Connotationally could work. [EDIT: Changed acronym to better suggestion by badger]

Footnotes

0: Anatoly Vorobey says in the comments that I'm using the word connotation too broadly. He suggests "subtext".

1: I feel like I might have seen this game on Overcoming Bias before, but I can't find it there. If I did, apologies to the original poster.

2: Comment with any other good ones you know.

3: Playing the Connotation Game a lot might also give you partial immunity to this.

4: This is a great example of a hotly-debated statement that is desperately in need of reduction.

Comment author: Eliezer_Yudkowsky 22 March 2009 05:13:13AM 17 points [-]

So... first of all, I'd like someone to look up the logical positivists and say what it is they actually believed. My impression is that so far as their verbal description of their philosophy went, if not its actual use, they claimed that the meaning of any phrase consisted entirely in its impact on experience, and that no other aspect of it is meaningful. This implies that a theory of photons which had photons vanishing as soon as they crossed the horizon of the expanding universe, and a theory which had the photons continuing undetectably onward, had the same meaning.

If this is not logical positivism, then let me be corrected.

The position you're describing sounds to me like what I would call reductionism, and I would agree with the caveat that certain meaningful entities can have logical elements - for example, I am willing to consider "the sum of 2 + 2" apart from any particular calculator that calculates it; its meaning is distinct from the meaning of "the result of calculator X" where calculator X is any physical thing I can point to including my own brain. I have no idea if this reflects reality, but I am unable to make my map work without logical as well as physical elements. I am, however, entirely willing to reduce every meaning to some mixture of physical stuffs and abstract computations.

Is there any point in arguing over whether we are "logical positivists" apart from the particulars of the stance? :)

Comment author: Yvain 22 March 2009 01:21:46PM *  6 points [-]

I'm hesitant to use "reductionism" because I already interpret that to be a belief about the material world (747s made of quarks and so on), not about propositions. I know people who accept material reductionism, but not propositional reductionism.

The real positivists were willing to accept that 2+2=4 was irreducible, since they considered it a tautology/definition and so exempt from testing. I am split: I think in one sense it's tautological, but that we pay attention to that particular tautology for reasons involving a testable generalization over all cases where two objects have been added to two objects and the result has been four objects.

Comment author: Eliezer_Yudkowsky 21 March 2009 07:44:33PM 28 points [-]

I should note that if I'm teaching deep things, then I view it as important to make people feel like they're learning deep things, because otherwise, they will still have a hole in their mind for "deep truths" that needs filling, and they will go off and fill their heads with complete nonsense that has been written in a more satisfying style.

Comment author: Yvain 22 March 2009 01:00:24PM *  9 points [-]

Please make a top-level post on this. Not because it needs any more explanation, but because everyone needs to see it, and I need a detailed and official-looking version of it to link all of my friends to (especially those who are teachers).

Comment author: Yvain 22 March 2009 12:42:03PM *  67 points [-]

I read recently an article on charitable giving which mentioned how people split up their money among many different charities to, as they put it, "maximize the effect", even though someone with this goal should donate everything to the single highest-utility charity. And this seems a bit like the example you cited where, if blue cards came up randomly 75% of the time and red cards came up 25% of the time, people would bet on blue 75% of the time even though the optimal strategy is blue 100%. All this seems to come from concepts like "Don't put all your eggs in one basket", which is a good general rule for things like investing but can easily break down.

I find myself having to fight this rule for a lot of things, and one of them is beliefs. If all of my opinions are Eliezer-ish, I feel like I'm "putting all my eggs in one basket", and I need to "diversify".You use book recommendations as a reductio, but I remember reading about half the books on your recommended reading list, thinking "Does reading everything off of one guy's reading list make me a follower?" and then thinking "Eh, as soon as he stops recommending such good books, I'll stop reading them."

The other thing is the Outside View summed up by the proverb "If two people think alike, one of them isn't thinking." In the majority of cases I observe where a person conforms to all of the beliefs held by a charismatic leader of a cohesive in-group, and keeps praising that leader's incredible insight, that person is a sheeple and that leader has a cult (see: religion, Objectivism, various political movements). I respect the Outside View enough that I have trouble replacing it with the Inside View that although I agree with Eliezer about nearly everything and am willing to say arbitrarily good things about him, I'm certainly not a cultist because I'm coming to my opinions based on Independent Logic and Reason. I don't know any way of solving this problem except the hard way.

"note: Hofstadter does not have a cult"

I tried to start a Hofstadter cult once. The first commandment was "Thou shalt follow the first commandment." The second commandment was "Thou shalt follow only those even-numbered commandments that do not exhort thee to follow themselves." I forget the other eight. Needless to say it didn't catch on.

Comment author: komponisto 22 March 2009 03:52:43AM 3 points [-]

Eliezer, in No Logical Positivist I, condemns the positivist notion of verifiability for excluding some perfectly meaningful propositions. For example, he says, it may be that a chocolate cake formed in the center of the sun on 8/1/2008, then disappeared after one second. ... But there's no way to test it ... so the logical positivists would dismiss it as nonsense.

As I recall, this interpretation was disputed in the comments on that post. In addition to referring readers there, let me also direct those interested in learning more about logical positivism to this interview with A.J. Ayer. If memory serves, he mentions espousing a view of ethical and aesthetic statements similar to Yvain's interpretation of statements like "Islam is a religion of peace" (often called emotivism, or the "boo-hurray theory").

Comment author: Yvain 22 March 2009 12:18:25PM 2 points [-]

I was one of the ones disputing in those comments, but I have to admit that g's "The absolute is an uncle" example stopped me short. I developed the "logs of the universe" theory in response to that, but I'm not sure how good a response it was - hence footnote 2, about how it was easy in practice but kind of hard in theory.

Emotivism is very relevant here and I was thinking of making a followup post on it. Good catch.

Comment author: byrnema 22 March 2009 04:05:06AM *  5 points [-]

I would like to give an example that I think fits here. (Do you think it fits?) It has to do with the common practice of summarizing the whole abortion debate with whether or not "life begins at conception". Putting the debate entirely aside (of course!) this is my favorite example of words that have no meaning outside of identifying yourself with this group or that group, and thus is the beginning and the end of terribly sloppy thinking.

Does life begin at conception? We need to define "life" first. OK, anything that is composed of cells and metabolizes is some sort of common definition. For simplicity, let's call the unit of life an "organism". So we are asking if an organism first begins living at conception. Sort of obviously "yes" and totally irrelevant when you think about what the words actually mean, yet somehow this frames the debate for many people.

Comment author: Yvain 22 March 2009 12:16:09PM 4 points [-]

I agree. This is part of the problem someone (maybe Phil Goetz?) was mentioning before about how our society has this very binary mode of thinking where taking life is always automatically bad. The solution here is taboo "life" and find the disguised query you're really wondering about. In my case, the question is whether the abortion causes suffering to an entity capable of feeling suffering. In the case of a fetus, arguably yes; in the case of an egg, very likely no.

Comment author: komponisto 21 March 2009 05:38:10AM 3 points [-]

In particular, poetry is a more constrained, more beauty-focused form of language, whereas abstract art and jazz and atonal jazz are less constrained, less beauty-focused forms of art and music.

Ouch. I'm generally a fan of your posts, Yvain, but this remark makes me wince.

To quote Milton Babbitt (regarding composers who claimed not to be "using a system"), ignorance of constraint does not imply absence of constraint. The particular constraints that are operative in jazz or abstract art may not be immediately apparent to the outsider, but that doesn't mean they're not there.

Also, to say that certain art forms are "less beauty-focused" comes dangerously close to a rhetorical shot. There are exceptions, but as a general rule it's safe to say that the practitioners of any art are seeking to create beautiful works. To a first approximation, art:beauty::rationality:truth.

Finally, let me caution everyone that "atonal" is a technical term that really ought not to be bandied around by people without a background in music. In particular, it does not mean "unfamiliar-sounding music I don't like", as it sometimes seems to in some quarters. (I'm not accusing anyone in particular of doing this, just a general warning.)

Comment author: Yvain 21 March 2009 09:15:18PM 2 points [-]

You've figured me out. I have no musical background whatsoever and am making judgments solely based on my superficial untrained perception. I should probably stop that.

I have heard many abstract artists condemn art that is too focused on beauty as naive and unworthy of a true artist, and support art that makes philosophical or political points instead, but I don't know for sure that all abstract artists are like this.

Comment author: Yvain 21 March 2009 08:59:48PM *  8 points [-]

Robin's post seemed to be about the marginal value of rationality. Being completely irrational is a one way ticket to death or ruin, I agree. But there are fewer ways to die by refusing to go from ordinary high-IQ university-educated person to person who has read and applied the Overcoming Bias techniques. They're still there, but they're not quite as obvious. Most of the ones I can think of involve medicine, and Robin probably disagrees and doesn't think those matter so much.

The Power of Positivist Thinking

68 Post author: Yvain 21 March 2009 08:55PM

Related to: No Logical Positivist I, Making Beliefs Pay Rent, How An Algorithm Feels From Inside, Disguised Queries

Call me non-conformist, call me one man against the world, but...I kinda like logical positivism.

The logical positivists were a dour, no-nonsense group of early 20th-century European philosophers. Indeed, the phrase "no-nonsense" seems almost invented to describe the Positivists. They liked nothing better then to reject the pet topics of other philosophers as being untestable and therefore meaningless. Is the true also the beautiful? Meaningless! Is there a destiny to the affairs of humankind? Meaningless? What is justice? Meaningless! Are rights inalienable? Meaningless!

Positivism became stricter and stricter, defining more and more things as meaningless, until someone finally pointed out that positivism itself was meaningless by the positivists' definitions, at which point the entire system vanished in a puff of logic. Okay, it wasn't that simple. It took several decades and Popper's falsifiabilism to seal its coffin. But vanish it did. It remains one of the least lamented theories in the history of philosophy, because if there is one thing philosophers hate it's people telling them they can't argue about meaningless stuff.

But if we've learned anything from fantasy books, it is that any cabal of ancient wise men destroyed by their own hubris at the height of their glory must leave behind a single ridiculously powerful artifact, which in the right hands gains the power to dispel darkness and annihilate the forces of evil.

The positivists left us the idea of verifiability, and it's time we started using it more.



Eliezer, in No Logical Positivist I, condemns the positivist notion of verifiability for excluding some perfectly meaningful propositions. For example, he says, it may be that a chocolate cake formed in the center of the sun on 8/1/2008, then disappeared after one second. This statement seems to be meaningful; that is, there seems to be a difference between it being true or false. But there's no way to test it (at least without time machines and sundiver ships, which we can't prove are possible) so the logical positivists would dismiss it as nonsense.

I am not an expert in logical positivism; I have two weeks studying positivism in an undergrad philosophy class under my belt, and little more. If Eliezer says that is how the positivists interpreted their verifiability criterion, I believe him. But it's not the way I would have done things, if I'd been in 1930s Vienna. I would have said that any statement corresponding to a state of the material universe, reducible in theory to things like quarks and photons, testable by a being who has access to the machine running the universe1 and who can check the logs at will - such a statement is meaningful2. In this case the chocolate cake example passes: it corresponds to a state of the material world, and is clearly visible on the universe's logs. "Rights are inalienable" remains meaningless, however. At the risk of reinventing the wheel3, I will call this interpretation "soft positivism".

My positivism gets even softer, though. Consider the statement "Google is a successful company." Though my knowledge of positivism is shaky, I believe that most positivists would reject this as meaningless; "success" is too fuzzy to be reduced to anything objective. But if positivism is true, it should add up to normality: we shouldn't find that an obviously useful statement like "Google is a successful company" is total nonsense. I interpret the statement to mean certain objectively true propositions like "The average yearly growth rate for Google has been greater than the average yearly growth rate for the average company", which itself reduces down to a question of how much money Google made each year, which is something that can be easily and objectively determined by anyone with the universe's logs.

I'm not claiming that "Google is a successful company" has an absolute one-to-one identity with a statement about average growth rates. But the "successful company" statement is clearly allied with many testable statements. Average growth rate, average profits per year, change in the net worth of its founders, numbers of employees, et cetera. Two people arguing about whether Google was a successful company could in theory agree to create a formula that captures as much as possible of their own meaning of the word "successful", apply that formula to Google, and see whether it passed. To say "Google is a successful company" reduces to "I'll bet if we established a test for success, which we are not going to do, Google would pass it."

(Compare this to Eliezer's meta-ethics, where he says "X is good" reduces to "I'll bet if we calculated out this gigantic human morality computation, which we are not going to do, X would satisfy it.")

This can be a very powerful method for resolving debates. I remember getting into an argument with my uncle, who believed that Obama's election would hurt America because having a Democratic president is bad for the economy. We were doing the normal back and forth, him saying that Democrats raised taxes which discouraged growth, me saying that Democrats tended to be more economically responsible and less ideologically driven, and we both gave lots of examples and we never would have gotten anywhere if I hadn't said "You know what? Can we both agree that this whole thing is basically asking whether average GDP is lower under Democratic than Republican presidents?" And he said "Yes, that's pretty much what we're arguing about." So I went and got the GDP statistics, sure enough they were higher under Democrats, and he admitted I had a point4.

But people aren't always as responsible as my uncle, and debates aren't always reducible to anything as simple as GDP. Consider: Zahra approaches Aaron and says: "Islam is a religion of peace."5

Perhaps Aaron disagrees with this statement. Perhaps he begins debating. There are many things he could say. He could recall all the instances of Islamic terrorism, he could recite seemingly violent verses from the Quran, he could appeal to wars throughout history that have involved Muslims. I've heard people try all of these.

And Zahra will respond to Aaron in the same vein. She will recite Quranic verses praising peace, and talk about all the peaceful Muslims who never engage in terrorism at all, and all of the wars started by Christians in which Muslims were innocent victims. I have heard all these too.

Then Paula the Positivist comes by. "Hey," she says, "We should reduce this statement to testable propositions, and then there will be no room for disagreement."

But maybe, if asked to estimate the percentage of Muslims who are active in terrorist groups, Aaron and Zahra will give the exact same number. Perhaps they are both equally aware of all the wars in history in which Muslims were either aggressors or peacemakers. They may both have the entire Quran memorized and be fully aware of all appropriate verses. But even after Paula has checked to make sure they agree on every actual real world fact, there is no guarantee that they will agree on whether Islam is a religion of peace or not.

What if we ask Aaron and Zahra to reduce "Islam is a religion of peace" to an empirical proposition? In the best case, they will agree on something easy, like "Muslims on average don't commit any more violent crimes than non-Muslims." Then you just go find some crime statistics and the problem is solved. In the second-best case, the two of them reduce it to completely different statements, like "No Muslim has ever committed a violent act" versus "Not all Muslims are violent people." This is still a resolution to the argument; both Aaron and Zahra may agree that the first proposition is false and the second proposition is true, and they both agree the original statement was too vague to go around professing.

In the worst-case scenario, they refuse to reduce the statement at all, or they deliberately reduce it to something untestable, or they reduce it to two different propositions but are outraged that their opponent is using a different proposition than they are and think their opponent's proposition is clearly not equivalent to the original statement.

How are they continuing to disagree, when they agree on all of the relevant empirical facts and they fully understand the concept of reducing a proposition?

In How an Algorithm Feels From the Inside, Eliezer writes about disagreement on definitions. "We know where Pluto is, and where it's going; we know Pluto's shape, and Pluto's mass - but is it a planet?" The question, he says, is meaningless. It's a spandrel from our cognitive algorithm, which works more efficiently if it assigns a separate central variable is_a_planet apart from all the actual tests that determine whether something is a planet or not.

Aaron and Zahra seem to be making the same sort of mistake. They have a separate variable is_a_religion_of_peace that's sitting there completely separate from all of the things you might normally use to decide whether one group of people is generally more violent than another.

But things get much worse than they do in the Pluto problem. Whether or not Pluto is a planet feels like a factual issue, but turns out to be underdetermined by the facts. Whether or not Islam is a religion of peace feels like a factual issue, but is really a false front for a whole horde of beliefs that have no relationship to the facts at all.

When Zahra says "Islam is a religion of peace," she is very likely saying something along the lines of "I like Islam!" or "I like tolerance!" or "I identify with an in-group who say things like 'Islam is a religion of peace'" or "People who hate Islam are mean!" or even "I don't like Republicans.". She may be covertly pushing policy decisions like "End the war on terror" or "Raise awareness of unfair discrimination against Muslims."

When Aaron says "Islam is not a religion of peace," he is probably saying something like "I don't like Islam," or "I think excessive tolerance is harmful", or "I identify with an in-group who would never say things like 'Islam is a religion of peace'" or even "I don't like Democrats." He may be covertly pushing policy decisions like "Continue the war on terror" or "Expel radical Muslims from society."

Eliezer's solution to the Pluto problem is to uncover the disguised query that made you care in the first place. If you want to know whether Pluto is spherical under its own gravity, then without worrying about the planet issue you can simply answer yes. And you're wondering whether to worry about your co-worker Abdullah bombing your office, you can simply answer no. Islam is peaceful enough for your purposes.

But although uncovering the disguised query is a complete answer to the Pluto problem, it's only a partial answer to the religion of peace problem. It's unlikely that someone is going to misuse the definition of Pluto as a planet or an asteroid to completely misunderstand what Pluto is or what it's likely to do (although it can happen). But the entire point of caring about the "Islam is a religion of peace" issue is so you can misuse it as much as possible.

Israel is evil, because it opposes Muslims, and Islam is a religion of peace. The Democrats are tolerating Islam, and Islam is not a religion of peace, so the Democrats must have sold out the country. The War on Terror is racist, because Islam is a religion of peace. We need to ban headscarves in our schools, because Islam is not a religion of peace.

I'm not sure how the chain of causation goes here. It could be (emotional attitude to Islam) -> (Islam [is/isn't] a religion of peace) -> (poorly supported beliefs about Islam). Or it could just be (emotional attitude to Islam) -> (poorly supported beliefs about Islam). But even in the second case, that "Islam [is/isn't] a religion of peace" gives the poorly supported beliefs a dignity that they would not otherwise have, and allows the person who holds them to justify themselves in an argument. Basically, that one phrase holes itself up in your brain and takes pot shots at any train of thought that passes by.

The presence of that extra is_a_religion_of_peace variable is not a benign feature of your cognitive process anymore. It's a malevolent mental smuggler transporting prejudices and strong emotions into seemingly reasonable thought processes.

Which brings us back to soft positivism. If we find ourselves debating statements that we refuse to reduce to empirical data6, or using statements in ways their reductions don't justify, we need to be extremely careful. I am not positivist enough to say we should never be doing it. But I think it raises one heck of a red flag.

Agree with me? If so, which of the following statements do you think are reducible, and how would you begin reducing them? Which are completely meaningless and need to be scrapped? Which ones raise a red flag but you'd keep them anyway?

1. All men are created equal.
2. The lottery is a waste of hope.
3. Religious people are intolerant.
4. Government is not the solution; government is the problem.
5. George Washington was a better president than James Buchanan.
6. The economy is doing worse today than it was ten years ago.
7. God exists.
8. One impulse from a vernal wood can teach you more of man, of moral evil, and of good than all the sages can.
9. Imagination is more important than knowledge.
10. Rationalists should win.

 

Footnotes:

1: More properly the machine running the multiverse, since this would allow counterfactuals to be meaningful. It would also simplify making a statement like "The patient survived because of the medicine", since it would allow quick comparison of worlds where the patient did and didn't receive it. But if the machine is running the multiverse, where's the machine?

2: One thing I learned from the comments on Eliezer's post is that this criterion is often very hard to apply in theory. However, it's usually not nearly as hard in practice.

3: This sounds like the sort of thing there should already be a name for, but I don't know what it is. Verificationism is too broad, and empiricism is something else. I should point out that I am probably misrepresenting the positivist position here quite badly, and that several dead Austrians are either spinning in their graves or (more likely) thinking that this whole essay is meaningless. I am using "positivist" only as a pointer to a certain style of thinking.

4: Before this issue dominates the comments thread: yes, I realize that the president having any impact on the economy is highly debatable, that there's not nearly enough data here to make a generalization, et cetera. But my uncle's statement - that Democratic presidents hurt the economy, is clearly not supported.

5: If your interpretation of anything in the following example offends you, please don't interpret it that way.

6: Where morality fits into this deserves a separate post.

Comment author: PhilGoetz 21 March 2009 04:20:32AM 5 points [-]

I don't understand -- are you claiming that scientists are people and therefore they're as much experts on ethics as anyone?

Yes. Actually, I would say scientists are better ethicists in their area of expertise, because

  • moral reasoning is reasoning, and smarter people are better at reasoning

  • they know what the heck they're talking about.

Current bioethicists may suck, but the idea of having some people specialize at bioethics seems sound.

Can you specialize in ethics? Or is it like - to use the ever-popular reason-as-martial-arts metaphor - like specializing in kata? You sometimes see schools that strongly emphasize kata. IMHO their kata is weak, because they don't understand the purpose of their movements. To answer to this question, you need to ask whether moral reasoning within a domain is qualitatively different from any other kind of reasoning in a domain.

Perhaps if our debates on ethics used esoteric concepts from category theory and the writings of German philosophers, it would be of some benefit to specialize in ethics. But they have never risen to that level.

Comment author: Yvain 21 March 2009 03:01:07PM 2 points [-]

I would trust someone who understood and could use utilitarianism to solve ethical issues better than someone who didn't. Of course, modern bioethicists don't, so this is hardly a point in their favor. But I think in a perfect world people could specialize in ethics and gain unusual competence in that field.

The one real worry I have about scientists is that they're too personally invested. I wouldn't trust the guy who'd spent ten years of his life inventing a stem cell technique to determine when the technique probably shouldn't be used because of ethical issues. And I think that carries over to entire fields; biologists, in general, will have an personal investment in biological discoveries.

Optimal solution is smart people with scientific training specializing in utilitarian ethics. In our own world, I trust scientists about as much as anyone else, maybe a little more.

Comment author: Yvain 20 March 2009 10:31:47PM *  16 points [-]

I agree with your conclusion. That said, your title is missing an "s".

Comment author: Yvain 20 March 2009 07:36:19PM *  6 points [-]

Wait a second, now we're using Jews trying to run a synagogue as an example of a group who cooperate and don't always disagree with each other for the sake of disagreeing? Your synagogue must have been very different from mine. You never heard the old "Ten Jews, ten opinions - or twenty if they're Reform" joke? Or the desert island joke?

I also agree with everyone. In particular, I agree with Cameron and Prase that it's tough to just say "I agree". I agree with ciphergoth that I worry that I'm sucking up to you too much. I agree with Anna Salamon that we tend to be intellectual show-offs. I agree with Julian that many of us probably started off with a contrarian streak and then became rationalists. I agree with Jacob Lyles that there's a strong game theory element here - I lose big if rationalists don't cooperate, I win a little if we all cooperate under Eliezer's benevolent leadership, but to a certain way of thinking I win even more if we all cooperate under my benevolent leadership and there's no universally convincing proof that cooperating under someone else is always the highest utility option. And I agree with practically everything in the main post.

One thing I don't agree with: being ashamed of strong feelings isn't a specifically rationalist problem. It's a broader problem with upper/middle class society. Possibly more on this later.

Comment author: Emile 20 March 2009 04:44:55PM 9 points [-]

Be careful about using the "rationalists should win" slogan too literally. Martial artists should win too, but that doesn't mean they should take an AK-47 to their next sparring match and blowing their opponent's face off. Martial artists place high value on winning honorably. I see no reason why we shouldn't emulate them.

I disagree. The problem with using dishonest rethoric to win in a debate isn't that it's winning dishonorably; it's that it's winning at the wrong game - on a game that you wouldn't consider the most important if you looked at it closely.

To continue with the martial arts analogy, imagine say a Chinese kung fu master in World War 2 Nanjing that knows that Japanese soldiers are coming over to kill off all of his family. Should he try to win the fight honorably? Or just try to win using every dirty trick in the book (including running away)? If he focuses on winning honorably, he's lost sight of his main goal (save his family) in favor of a secondary one (win honorably).

Similarly, if you foxus on "winning the debate", and as a result push people into a corner that will make them dislike you and become more attached to their identity as a believer in whatever - you focused on the wrong subgoal, and lost at the one which was important to you.

Comment author: Yvain 20 March 2009 07:08:23PM 6 points [-]

I'm a precedent utilitarian. I try to maximize utility, except when doing so would set a bad precedent that would lower utility later.

Precedent utilitarians are usually good about restraining from force. Yes, killing a rich miser and distributing her money to the poor might increase utility. But it sets the precedent that anyone can kill someone if they think of a good enough reason, and most people won't be smart enough to limit themselves to genuinely good reasons. Therefore, precedent utilitarians generally respect the rule of not killing others. But in certain cases this rule breaks down. In the WWII example you mention, it doesn't seem particularly dangerous to set the precedent that you can use force against invaders coming to kill your family.

I try to use the same thought process when evaluating when to use rhetoric. If anyone can use rhetoric any time it furthers a goal that they consider genuinely good, then there's little incentive to use rational argument except on the rare hard-core rationalists who are mostly resistant to rhetorical tricks. I want to be able to condemn a demagogue who uses rhetoric without being a hypocrite. If I needed to use rhetoric in a situation where I couldn't blame anyone else for using rhetoric, like trying to save my family, I'd do it.

(the problem with precedent utilitarianism is that the calculations are impossible to do with real math, and mostly just involve handwaving. But I hope it at least gives a sketch of my thought processes)

Comment author: pjeby 19 March 2009 10:39:56PM 3 points [-]

Um, isn't it kind of rhetorical to compare rhetoric to force and murder?

Also, all your articles here that I recall -- likewise those of Eliezer on Overcoming Bias -- are masterful applications of rhetoric. So I'm kind of confused here. Is this one of those "do as I say, not as I do" things?

Comment author: Yvain 20 March 2009 06:59:20PM 5 points [-]

If you mean the articles here are clear or well argued, thank you. I have no objection to clarity or good argument; see the first paragraph of the comment above. If you mean that I'm using dirty tricks like the "terrorists win" example, then I'd like to know exactly what you mean so I can avoid doing it in the future.

When I compare rhetoric (meaning "empty rhetoric", as mentioned) to force and murder, I'm not saying they're equally bad, or doing one leads to the other or anything like that. Just that they're bad for the same reason. Both are potentially "useful" techniques. But both prevent rational argument and if used too frequently lead to a world in which rational argument is impossible.

Comment author: John_Maxwell_IV 20 March 2009 05:05:48AM 1 point [-]

Poetry's relationship to language seems analogous to atonal jazz's relationship to music, and abstract art's relationship to art. I'm not a fan of any of those (although I do enjoy the occasional weird progressive rock song), but it would be interesting to see how well liking one predicts liking another, or how well intelligence in a domain predicts interest in its more esoteric works.

Comment author: Yvain 20 March 2009 01:53:39PM 3 points [-]

I like poetry but dislike jazz and abstract art.

I agree with Michael that you may be confusing poetry with modernist poetry. In particular, poetry is a more constrained, more beauty-focused form of language, whereas abstract art and jazz and atonal jazz are less constrained, less beauty-focused forms of art and music.

My analogy would be modernist poetry: poetry :: abstract art: art.

Comment author: Z_M_Davis 20 March 2009 04:37:45AM *  2 points [-]

Thus John Keats, in the same year he wrote Lamia, also penned perhaps the greatest statement of the Joy in the Merely Real ideal ever, writing: "Beauty is Truth, Truth Beauty, that is all / Ye know on Earth, and all ye need to know."

Maybe I'm interpreting it wrong, but I've never liked that couplet (likewise Dickinson's expression of the same sentiment). Truth is not beauty! We're the products of billions of years of bloody blind natural selection, clinging desperately to what scraps of value we've managed to achieve in this cold, uncaring universe! Everything you care about, everything you believe in (in the comforting, nonrationalist sense) means shit to the Price equation! It's this sentiment that makes me like the poem preceding Greg Egan's Distress, which ends:

And there must be room for all at the celebration of understanding

for there is a truth which cannot be bought or sold

imposed by force, resisted

or escaped.

That's more like it. By all means, appreciate the beauty of the natural world---but not so much that it doesn't scare you. Rationality is beautiful, but the truth is just the truth.

Other nominations ... maybe Sir Walter Raleigh's "The Lie"? (Note that "give them the lie" here is an idiom for "Tell them that they're lying.") My favorite poem is Stephen Spender's "I Think Continually of Those Who Were Truly Great," but it might be stretching the evidence to call it rationality-related---and of course stretching the evidence is something we must not do.

Comment author: Yvain 20 March 2009 01:49:11PM *  2 points [-]

Beauty is not always nice or friendly. There is greater beauty in the strike of a cobra than...

...sorry, I'm still thinking in poetry mode. Yes, in any reasonable interpretation, "truth is beauty" is completely false (counterexample: New Jersey exists and Minas Tirith doesn't). This is one of those cases where I am tempted to join the Dark Side and start saying things like "It's not true on a merely factual level, but..."

I think of those lines sort of the same way I think of Buckminster Fuller saying "When I am working on a problem I never think about beauty. I only think about how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." Or Pablo Picasso saying "Art is the lie that reveals the truth".

That there is a beauty to the structure of reality, and that bringing my beliefs closer to the truth makes me better able to partake in and increase that beauty, is one of the reasons this whole rationalism thing seems worth it to me. That's how I interpret Keats, at least. Who knows what he was thinking? Evidently not enough to abandon his grudge against science for proving there were no gnomes in mines.

...but I promise I won't start trying to use "Beauty is truth, therefore..." in any deductive arguments or anything.

Comment author: MBlume 20 March 2009 03:48:25AM *  4 points [-]

I think given an hour to talk to him and set him straight I could've convinced him there is no loss of beauty in accepting Newton's optics. It is true, after all.

I've often felt the same about C.S. Lewis. It is a peculiarity of the rationalist mode of thought that this is an expression of respect.

Comment author: Yvain 20 March 2009 01:14:31PM 2 points [-]

I also have great respect for C. S. Lewis...but I don't think an hour would be quite sufficient for him :)

Rationalist Poetry Fans, Unite!

32 Post author: Yvain 20 March 2009 01:58AM

Related to: Little Johnny Bayesian, Savanna Poets

There are certain stereotypes about what rationalists can talk about versus what's really beyond the pale. So far, Less Wrong has pretty consistently exploded those stereotypes. In the past three weeks, we've discussed everything from Atlantis to chaos magick to "9-11 Truth". But I don't think anything surprised me quite as much as learning that there are a couple of rationalists here with a genuine interest in poetry.

Poetry has not been very friendly to the rational worldview over the past few centuries. What with all the 19th century's talk of unweaving rainbows and the 20th century's talk of quadrupeds swooning into billiard balls, it's tempting to think it reflects some natural order of things, some eternal conflict between Art and Science.

But for most of human history, science and art were considered natural allies. Lucretius' De Rerum Natura, an argument for atheism and atomic theory famous for being the ancient Roman equivalent of The God Delusion, was written in poetry. All through the Middle Ages, artists worked to a philosophy of trying to depict and celebrate natural truth. And the eighteenth century saw a golden age of what was sometimes called "rationalist poetry", a versified celebration of Enlightenment principles.

When William Wordsworth launched his poetic jihad against rationalism, he called his declaration of war The Tables Turned. On a mundane level, the title referred to an argument he was having with his friend, but on a grander scale he was consciously inverting the previous order of Reason as the virtue of poetry. Thus:

Enough of Science and of Art;
Close up these barren leaves;
Come forth, and bring with you a heart
That watches and receives.

Over the next few years, he and fellow jihadis John Keats and Percy Bysshe Shelley were wildly successful in completely changing the poetic ideal. I can't begrudge them their little movement; their poetry ranks among the greatest art ever produced by humankind. But it bears repeating that there was a strong rationalist tradition in poetry before, during, and after the Romantic Era. In its honor, I thought I would share some of my favorite rationalist poems. I make no claims that this is exhaustive, representative, or anything else besides my personal choices.

The most famous rationalist poet is probably Alexander Pope (1688-1744), perhaps best known for writing Isaac Newton's epitaph:

Nature and nature's laws lay hid in night.
God said, "Let Newton be!" and all was light.1

Indeed, Pope spent much of his career praising science and human reason, while also simultaneously lampooning human stupidity:

Go, wondrous creature! mount where Science guides;
Go measure earth, weigh air, and state the tides;
Instruct the planets in what orbs to run,
Correct old Time, and regulate the sun;
Go, soar with Plato to th'empyreal sphere,
To the first good, first perfect, and first fair;
Or tread the mazy round his followers trod,
And quitting sense call imitating God;
As eastern priests in giddy circles run,
And turn their heads to imitate the sun.
Go, teach Eternal Wisdom how to rule--
Then drop into thyself, and be a fool!

I can't claim this as a complete victory for rationalism, since it was in the context of Essay on Man, a mysterianist work declaring that humans should never overreach their pathetic mental powers and question God's supremacy. Even the quoted passage is a little ironic, intended to convey that humankind, with such amazing science, had a tendency to shoot itself in the foot when it tried to overstep its bounds.

But Pope's appreciation for scientific progress was genuine, and he was also deeply interested in overcoming bias (which he, in his pre-Samuel Johnson way, called "byass"). His Essay on Criticism sometimes reads like a strangely spelled, classical-allusion-laden rationalists' manual:

Of all the Causes which conspire to blind
Man's erring Judgment, and misguide the Mind,
What the weak Head with strongest Byass rules,
Is Pride, the never-failing Vice of Fools [...]
If once right Reason drives that Cloud away,
Truth breaks upon us with resistless Day;
Trust not your self; but your Defects to know,
Make use of ev'ry Friend--and ev'ry Foe.

He exhorts us to think for ourselves, rather than take things on faith or blindly accept authority:

Some ne'er advance a Judgment of their own,
But catch the spreading Notion of the Town;
They reason and conclude by Precedent,
And own stale Nonsense which they ne'er invent.

But equally he reminds us that reversed stupidity is not intelligence:

The Vulgar thus through Imitation err;
As oft the Learn'd by being Singular;
So much they scorn the Crowd, that if the Throng
By Chance go right, they purposely go wrong;

And tells us to admit our errors, learn from them, and move on:

Some positive persisting Fops we know,
Who, if once wrong, will needs be always so;
But you, with Pleasure own your Errors past,
An make each Day a Critick on the last.

At the end, he describes the person he wants judging his poetry: someone who sounds rather like the ideal rationalist.

Unbiass'd, or by Favour or by Spite;
Not dully prepossest, nor blindly right;
Tho' Learn'd well-bred; and tho' well-bred, sincere;
Modestly bold, and Humanly severe
Who to a Friend his Faults can freely show,
And gladly praise the Merit of a Foe
Blest with a Taste exact, yet unconfin'd;
A Knowledge both of Books and Humankind;
Gen'rous Converse; a Sound exempt from Pride;
And Love to Praise, with Reason on his Side.

Pope came from a time when any person of good breeding was expected to be learned and able to converse about the scientific discoveries going on around them; an age when reason was actually trendy. There have been few such ages, and hence few such poets as Pope. But other rationalist poetry has come from people who were mathematicians or scientists in their day jobs, and poets only in their spare time.

Such a man was Omar Khayyam, the eleventh century Persian mathematician and astronomer. He did some work on cubic equations, wrote the Islamic world's most influential treatise on algebra, reformed the Persian calendar, and developed a partial heliocentric theory centuries before Copernicus. But he is most beloved for his rubaiyat, or quatrains, which recommend ignoring religion, accepting the deterministic material universe, and abandoning moral prudery in favor of having fun.

There are some beautiful translations and some accurate translations of Khayyam's works, but the rumor among those who speak Persian is that the beautiful translations are not accurate and the accurate translations are not beautiful, and that capturing the true spirit of the original may be hopeless. FitzGerald in particular, the most famous English translator, is accused of playing up the hedonism and playing down the rationalism. I've tried to select from a few different translations for this essay.

On determinism:

The Moving Finger writes; and, having writ,
Moves on: nor all thy Piety nor Wit
Shall lure it back to cancel half a Line,
Nor all thy Tears wash out a Word of it.

And that inverted Bowl we call The Sky,
Whereunder crawling coop't we live and die,
Lift not thy hands to It for help - for It
Rolls impotently on as Thou or I.

With Earth's first Clay They did the Last Man knead,
And there of the Last Harvest sow'd the Seed:
And the first Morning of Creation wrote
What the Last Dawn of Reckoning shall read.

On atheism:

What! out of senseless Nothing to provoke
A conscious Something to resent the yoke
Of unpermitted Pleasure, under pain
Of Everlasting Penalties, if broke!

What! from his helpless Creature be repaid
Pure Gold for what he lent him dross-allay'd--
Sue for a Debt he never did contract,
And cannot answer--Oh, the sorry trade!

In every step I take Thou sett'st a snare,
Saying,”I will entrap thee, so beware!"
And, while all things are under Thy command,
I am a rebel - as Thou dost declare.

On Joy in the Merely Real:

If in the Spring, she whom I love so well
Meet me by some green bank - the truth I tell -
Bringing my thirsty soul a cup of wine,
I want no better Heaven, nor fear a Hell.

And unlike Alexander Pope, who is horrified, HORRIFIED at the thought that mankind might challenge God's divine plan, Omar Khayyam thinks he could do better:

Ah, Love! could you and I with Him conspire
To grasp this sorry Scheme of Things entire,
Would not we shatter it to bits--and then
Re-mould it nearer to the Heart's Desire!

Needless to say, his contemporaries shunned him for such blasphemies. What would he say, they ask, when called before the throne of Allah to account for his beliefs? Well, he told them, he would say this:

Although I have not served Thee from my youth,
And though my face is mask'd with Sin uncouth,
In Thine Eternal Justice I confide,
As one who ever sought to follow Truth.

Compare the clarity of Khayyam, who is prepared to stand before God and justify himself without fear, to Pascal, who insists that we abandon our own intellectual integrity on the imperceptibly tiny chance that we might accrue some material gain. I find this quatrain - "in thy eternal justice I confide, as one who ever sought to follow Truth" - the only fully satisfying answer to Pascal's Wager.

Piet Hein (whom I've quoted here before) was another scientist who turned to poetry. During his career as a theoretical physicist and mathematician, he developed the superellipse and the game Hex (later studied by John Nash). His career as a poet began when the Nazis invaded his native Denmark. The censors would have prohibited any obviously rebellious literature, so he turned to writing odd little poems that seemed innocuous until you thought about them long enough, at which point they became obvious critiques of dictatorship. He continued writing after the war, usually on the theme of keeping things simple and avoiding stupidity.

This, for example, seems appropriate to a site called Less Wrong:

The road to wisdom? -- Well, it's plain
and simple to express:
      Err
      and err
      and err again
      but less
      and less
      and less.

If you've read Are You A Solar Deity or Schools Proliferating Without Evidence, you may see the humor in this quatrain about fitting the data to the theory:

Everything's either
concave or convex,
so whatever you dream
will be something with sex.

On the first virtue:

I'd like to know
what this whole show
is about
before it's out.

On the fifth virtue:

Truth shall emerge from the interplay
   of attitudes freely debated.
Don't be mislead by fanatics who say
   that only one truth should be stated:
truth is constructed in such a way
   that it can't be exaggerated.

On making an extraordinary effort:

Our so-called limitations, I believe,
apply to faculties we don't apply.
We don't discover what we can't achieve
until we make an effort not to try.

On fake justifications:

In view of your manner
    of spending your days
I hope you may learn,
    before ending them,
that the effort you spend
    on defending your ways
could better be spent
    on amending them.

Appropriate to the Singularity or to any of a number of fields:

Eradicate the optimist
 who takes the easy view
that human values will persist
 no matter what we do.

Annihilate the pessimist
 whose ineffectual cry
is that the goal's already missed
 however hard we try.

On reversed stupidity:

For many system shoppers it's
 a good-for-nothing system
that classifies as opposites
 stupidity and wisdom.

because by logic-choppers it's
 accepted with avidity:
stupidity's true opposite's
 the opposite stupidity.

On shutting up and doing the impossible:

'Impossibilities' are good
 not to attach that label to;
since, correctly understood,
if we wanted to, we would
 be able to be able to.

And even some poets who had no such formal acquaintance with science considered their poetry allied with its goals: an attempt to explore the universe and celebrate its wonders. This one's from Don Juan by Lord Byron, commonly (but, according to his own protestations, erroneously) classed with Wordsworth as a Romantic. I won't say there's not sarcasm in there, but Byron has a way of being sarcastic even when saying things he believes:

When Newton saw an apple fall, he found
In that slight startle from his contemplation --
'Tis said (for I'll not answer above ground
For any sage's creed or calculation) --
A mode of proving that the earth turn'd round
In a most natural whirl, called "gravitation;"
And this is the sole mortal who could grapple,
Since Adam, with a fall or with an apple.

Man fell with apples, and with apples rose,
If this be true; for we must deem the mode
In which Sir Isaac Newton could disclose
Through the then unpaved stars the turnpike road,
A thing to counterbalance human woes:
For ever since immortal man hath glow'd
With all kinds of mechanics, and full soon
Steam-engines will conduct him to the moon.

And wherefore this exordium? -- Why, just now,
In taking up this paltry sheet of paper,
My bosom underwent a glorious glow,
And my internal spirit cut a caper:
And though so much inferior, as I know,
To those who, by the dint of glass and vapour,
Discover stars and sail in the wind's eye,
I wish to do as much by poesy.

There's no sarcasm at all in this next declaration of Byron's, where he vows hostility to everything from despotism to religion to mob rule to fuzzy thinking to the Blue vs. Green two-party swindle:

And I will war, at least in words (and -- should
My chance so happen -- deeds), with all who war
With Thought; -- and of Thought's foes by far most rude,
Tyrants and sycophants have been and are.
I know not who may conquer: if I could
Have such a prescience, it should be no bar
To this my plain, sworn, downright detestation
Of every depotism in every nation.

It is not that I adulate the people:
Without me, there are demagogues enough,
And infidels, to pull down every steeple,
And set up in their stead some proper stuff.
Whether they may sow scepticism to reap hell,
As is the Christian dogma rather rough,
I do not know; -- I wish men to be free
As much from mobs as kings -- from you as me.

The consequence is, being of no party,
I shall offend all parties: never mind!
My words, at least, are more sincere and hearty
Than if I sought to sail before the wind.
He who has nought to gain can have small art: he
Who neither wishes to be bound nor bind,
May still expatiate freely, as will I,
Nor give my voice to slavery's jackal cry.

Byron on the progress of science, and on rejecting disproven theories:

If from great nature's or our own abyss
Of thought we could but snatch a certainty,
Perhaps mankind might find the path they miss --
But then 't would spoil much good philosophy.
One system eats another up, and this
Much as old Saturn ate his progeny;
For when his pious consort gave him stones
In lieu of sons, of these he made no bones.

But System doth reverse the Titan's breakfast,
And eats her parents, albeit the digestion
Is difficult. Pray tell me, can you make fast,
After due search, your faith to any question?
Look back o'er ages, ere unto the stake fast
You bind yourself, and call some mode the best one.
Nothing more true than not to trust your senses;
And yet what are your other evidences?

Again on the same topic (and some thoughts on the "wisdom of crowds"):

There is a common-place book argument,
Which glibly glides from every tongue;
When any dare a new light to present,
"If you are right, then everybody 's wrong"!
Suppose the converse of this precedent
So often urged, so loudly and so long;
"If you are wrong, then everybody's right"!
Was ever everybody yet so quite?

This is Byron at his sarcastic best on the value accorded truth in society:

The antique Persians taught three useful things,
To draw the bow, to ride, and speak the truth.
This was the mode of Cyrus, best of kings --
A mode adopted since by modern youth.
Bows have they, generally with two strings;
Horses they ride without remorse or ruth;
At speaking truth perhaps they are less clever,
But draw the long bow better now than ever.

I can't help ending this by saying a word in praise of the Romantics. Yes, they may have gotten their rainbows in a tangle, and they may have hurled every curse they could at "Reason", but I think they were less opposed than they let on. Consider as anecdotal evidence Percy Shelley, who was expelled from Oxford after refusing to recant his atheism. What the Romantics hated was anyone telling them how to think, and their quarrel with a science they did not understand was less with its methods and more that it seemed an authority. Thus John Keats, in the same year he wrote Lamia, also penned perhaps the greatest statement of the Joy in the Merely Real ideal ever, writing:

Beauty is Truth, Truth Beauty, that is all
Ye know on Earth, and all ye need to know.

I think given an hour to talk to him and set him straight I could've convinced him there is no loss of beauty in accepting Newton's optics. It is true, after all.

I end with Shelley's description from Mont Blanc of the godless yet ordered essence of the universe that he worshipped:

The secret Strength of things
Which governs thought, and to the infinite dome
Of Heaven is as a law.

The laws that govern our own thought processes are the same laws that bind the infinite dome of Heaven. What better statement of the rationalist worldview could you ask for?

Now, what are your favorite rationalist poems?

Footnotes:

1: I have since spotted the following addition to Pope's couplet:

It did not last; the Devil, howling "Ho!
Let Einstein be!" restored the status quo.

How to Not Lose an Argument

109 Post author: Yvain 19 March 2009 01:07AM

Related to: Leave a Line of Retreat

Followup to: Talking Snakes: A Cautionary Tale, The Skeptic's Trilemma

"I argue very well. Ask any of my remaining friends. I can win an argument on any topic, against any opponent. People know this, and steer clear of me at parties. Often, as a sign of their great respect, they don't even invite me."

        --Dave Barry

The science of winning arguments is called Rhetoric, and it is one of the Dark Arts. Its study is forbidden to rationalists, and its tomes and treatises are kept under lock and key in a particularly dark corner of the Miskatonic University library. More than this it is not lawful to speak.

But I do want to talk about a very closely related skill: not losing arguments.

Rationalists probably find themselves in more arguments than the average person. And if we're doing it right, the truth is hopefully on our side and the argument is ours to lose. And far too often, we do lose arguments, even when we're right. Sometimes it's because of biases or inferential distances or other things that can't be helped. But all too often it's because we're shooting ourselves in the foot.

How does one avoid shooting one's self in the foot? In rationalist language, the technique is called Leaving a Social Line of Retreat. In normal language, it's called being nice.

First, what does it mean to win or lose an argument? There is an unspoken belief in some quarters that the point of an argument is to gain social status by utterly demolishing your opponent's position, thus proving yourself the better thinker. That can be fun sometimes, and if it's really all you want, go for it.

But the most important reason to argue with someone is to change his mind. If you want a world without fundamentalist religion, you're never going to get there just by making cutting and incisive critiques of fundamentalism that all your friends agree sound really smart. You've got to deconvert some actual fundamentalists. In the absence of changing someone's mind, you can at least get them to see your point of view. Getting fundamentalists to understand the real reasons people find atheism attractive is a nice consolation prize.

I make the anecdotal observation that a lot of smart people are very good at winning arguments in the first sense, and very bad at winning arguments in the second sense. Does that correspond to your experience?

Back in 2008, Eliezer described how to Leave a Line of Retreat. If you believe morality is impossible without God, you have a strong disincentive to become an atheist. Even after you've realized which way the evidence points, you'll activate every possible defense mechanism for your religious beliefs. If all the defense mechanisms fail, you'll take God on utter faith or just believe in belief, rather than surrender to the unbearable position of an immoral universe.

The correct procedure for dealing with such a person, Eliezer suggests, isn't to show them yet another reason why God doesn't exist. They'll just reject it along with all the others. The correct procedure is to convince them, on a gut level, that morality is possible even in a godless universe. When disbelief in God is no longer so terrifying, people won't fight it quite so hard and may even deconvert themselves.

But there's another line of retreat to worry about, one I experienced firsthand in a very strange way. I had a dream once where God came down to Earth; I can't remember exactly why. In the borderlands between waking and sleep, I remember thinking: I feel like a total moron. Here I am, someone who goes to atheist groups and posts on atheist blogs and has told all his friends they should be atheists and so on, and now it turns out God exists. All of my religious friends whom I won all those arguments against are going to be secretly looking at me, trying as hard as they can to be nice and understanding, but secretly laughing about how I got my comeuppance. I can never show my face in public again. Wouldn't you feel the same?

And then I woke up, and shook it off. I am an aspiring rationalist: if God existed, I would desire to believe that God existed. But I realized at that point the importance of the social line of retreat. The psychological resistance I felt to admitting God's existence, even after having seen Him descend to Earth, was immense. And, I realized, it was exactly the amount of resistance that every vocally religious person must experience towards God's non-existence.

There's not much we can do about this sort of high-grade long-term resistance. Either a person has enough of the rationalist virtues to overcome it, or he doesn't. But there is a less ingrained, more immediate form of social resistance generated with every heated discussion.

Let's say you approach a theist (let's call him Theo) and say "How can you, a grown man, still believe in something stupid like talking snakes and magic sky kings? Don't you know you people are responsible for the Crusades and the Thirty Years' War and the Spanish Inquisition? You should be ashamed of yourself!"

This suggests the following dichotomy in Theo's mind: EITHER God exists, OR I am an idiot who believes in stupid childish  things and am in some way partly responsible for millions of deaths and I should have lower status and this arrogant person who's just accosted me and whom I already hate should have higher status at my expense.

Unless Theo has attained a level of rationality far beyond any of us, guess which side of that dichotomy he's going to choose? In fact, guess which side of that dichotomy he's now going to support with renewed vigor, even if he was only a lukewarm theist before? His social line of retreat has been completely closed off, and it's your fault.

Here the two definitions of "winning an argument" I suggested before come into conflict. If your goal is to absolutely demolish the other person's position, to make him feel awful and worthless - then you are also very unlikely to change his mind or win his understanding. And because our culture of debates and mock trials and real trials and flaming people on Usenet encourages the first type of "winning an argument", there's precious little genuine mind-changing going on.

Really adjusting to the second type of argument, where you try to convince people, takes a lot more than just not insulting people outright1. You've got to completely rethink your entire strategy. For example, anyone used to the Standard Debates may already have a cached pattern of how they work. Activate the whole Standard Debate concept, and you activate a whole bunch of related thoughts like Atheists As The Enemy, Defending The Faith, and even in some cases (I've seen it happen) persecution of Christians by atheists in Communist Russia. To such a person, ceding an inch of ground in a Standard Debate may well be equivalent to saying all the Christians martyred by the Communists died in vain, or something similarly dreadful.

So try to show you're not just starting Standard Debate #4457. I remember once, during the middle of a discussion with a Christian, when I admitted I really didn't like Christopher Hitchens. Richard Dawkins, brilliant. Daniel Dennett, brilliant. But Christopher Hitchens always struck me as too black-and-white and just plain irritating. This one little revelation completely changed the entire tone of the conversation. I was no longer Angry Nonbeliever #116. I was no longer the living incarnation of All Things Atheist. I was just a person who happened to have a whole bunch of atheist ideas, along with a couple of ideas that weren't typical of atheists. I got the same sort of response by admitting I loved religious music. All of a sudden my friend was falling over himself to mention some scientific theory he found especially elegant in order to reciprocate2. I didn't end up deconverting him on the spot, but think he left with a much better appreciation of my position.

All of these techniques fall dangerously close to the Dark Arts, so let me be clear: I'm not suggesting you misrepresent yourself just to win arguments. I don't think misrepresenting yourself would even work; evolutionary psychology tells us humans are notoriously bad liars. Don't fake an appreciation for the other person's point of view, actually develop an appreciation for the other person's point of view. Realize that your points probably seem as absurd to others as their points seem to you. Understand that many false beliefs don't come from simple lying or stupidity, but from complex mixtures of truth and falsehood filtered by complex cognitive biases. Don't stop believing that you are right and they are wrong, unless the evidence points that way. But leave it at them being wrong, not them being wrong and stupid and evil.

I think most people intuitively understand this. But considering how many smart people I see shooting their own foot off when they're trying to convince someone3, some of them clearly need a reminder.

 

Footnotes

1: An excellent collection of the deeper and most subtle forms of this practice of this sort can be found in Dale Carnegie's How to Win Friends and Influence People, one of the only self-help books I've read that was truly useful and not a regurgitation of cliches and applause lights. Carnegie's thesis is basically that being nice is the most powerful of the Dark Arts, and that a master of the Art of Niceness can use it to take over the world. It works better than you'd think.

2: The following technique is definitely one of the Dark Arts, but I mention it because it reveals a lot about the way we think: when engaged in a really heated, angry debate, one where the insults are flying, suddenly stop and admit the other person is one hundred percent right and you're sorry for not realizing it earlier. Do it properly, and the other person will be flabbergasted, and feel deeply guilty at all the names and bad feelings they piled on top of you. Not only will you ruin their whole day, but for the rest of time, this person will secretly feel indebted to you, and you will be able to play with their mind in all sorts of little ways.

3: Libertarians, you have a particular problem with this. If I wanted to know why I'm a Stalin-worshipper who has betrayed the Founding Fathers for personal gain and is controlled by his base emotions and wants to dominate others by force to hide his own worthlessness et cetera, I'd ask Ann Coulter. You're better than that. Come on. And then you wonder why people never vote for you.

Comment author: AndySimpson 19 March 2009 06:23:05AM 12 points [-]

Rationalists should shouldn't deny themselves the utility of rhetoric. Any rational rationalist can see that rhetoric is the path to winning, a kind of social theatre that lubricates decision-making with irrational or intermittently rational groups. If a group needs to be convinced of a position within a finite amount of time, bare reasoning isn't always the best option.

Maybe that is too Machiavellian to be "really" rational, but it is the winning path.

Comment author: Yvain 19 March 2009 03:42:44PM *  10 points [-]

I think I am using "rhetoric" in a different way than Aristotle. For Aristotle, it was the art of speaking clearly and eloquently to communicate a position. I am using it more in the way people use when they say "empty rhetoric" or "political rhetoric". "Unless you give up your rights, the terrorists have already won" is my idea of an archetypal rhetorical technique. That may not be fair to the field of rhetoric, but I need some word to describe it and I can't think of a better one, so "rhetoric" it is.

Rhetoric is a technique that may be useful to rationalists, but it's not a rationalist technique. Compare the use of force. I may, as a rationalist, decide the best way towards my goal is murdering all who oppose me, in which case I'll want to know techniques like how to use an assault weapon. But there's still something fundamentally shady about the technique of killing people; it may just barely be justified on utilitarian grounds for a sufficiently important goal, but it's one of those things that you use only as a last resort and even then only after agonizing soul-searching. I feel confident saying that the technique of murdering people effectively as a Dark Art.

I feel the same way about rhetoric (by my pessimistic definition). Tricking people into believing things they have no legitimate evidence for can certainly be helpful, but the more people do it the worse the world gets. Not only do people end up with less than maximally accurate beliefs, but every rhetorician needs to promote Dark Side Epistemology in order to keep zir job. And if I use rhetoric, you need to start using rhetoric just to keep up, and sooner or later everyone's beliefs are completely skewed and inaccurate. It's not quite as Dark an Art as force is, and it's much easier to justify, but it's in the same category.

Be careful about using the "rationalists should win" slogan too literally. Martial artists should win too, but that doesn't mean they should take an AK-47 to their next sparring match and blowing their opponent's face off. Martial artists place high value on winning honorably. I see no reason why we shouldn't emulate them.

Comment author: Eliezer_Yudkowsky 19 March 2009 01:43:10AM 3 points [-]

I thought so too, but I wasn't an expert - it could have been some meter I didn't understand.

However, given that we have the core of the story structure here, someone can go through and fix the prosody and then we'll have a real rationalist children's rhyme!

PS: Who the heck is downvoting this?!

Comment author: Yvain 19 March 2009 03:13:40PM *  10 points [-]

"Someone can go through and fix the prosody."
Do you consider this a friendly amendment?

Little Johnny Bayesian thought he was very bright
But the other schoolkids mocked him every time he came in sight
He could count and sing and read; he could spell and guess the weather
Until one day, Big Bill told him real bright boys could grow a feather.

“Ach” he cried, his spirits sinking, “Could a thing like that be true
“That would mean I must be dumb, and being stupid makes me blue!”
He went home and massaged his scalp until it hurt all over
Till at last he felt a growth shoot out, just like a little clover

“It's true!” he cried with glee, “It's true, and I've got feathers sprouting”
His intelligence now certain, he grew calm and stopped his pouting
He ran to show his mother that he hadn't been all wrong
When he heard on television: “Real bright children's legs grow long.”

So he strained for days and weeks and months to try to get the proof
Worried sick about his brightness and remaining quite aloof
Till at last he took his measurement, and it had been a cinch
In the days he had been fretting he had grown his legs an inch!

So he skipped to school, but as he went a scientist passed by
And he muttered that his research showed that real bright boys could fly
And Johnny thought of what he'd learned, his feathers and his height
And in a flash, he grew convinced he had the gift of flight.

Little Johnny spent a while planning his display
He thought he'd stand upon a wall, and proudly he would say
Behold my feathers! See my legs! I've evidence I'm bright!
And dazzle all his friends by soaring off and taking flight.

He climbed the wall and smiled and he proudly gave his speech
He took a flying leap – and there his plan stopped with a screech
Instead of soaring throught the skies, he heard an awful SMACK
And looked up, dazed, to find his leg all bloody, blue, and black

His classmates started laughing, and he struggled to explain
The logic that implied he had a more than normal brain
His friends, his favorite TV show, the scientists - all knew
That he was smart, and if they said it, surely it was true?

They laughed and said “You're just too eager to believe, you fool
Your feathers are just hair, and boys grow taller as a rule
If what you heard was really true, you'd fly without a doubt
But putting garbage in your logic gets you garbage out.

So Johnny made a bad mistake and ended with a cast
He'd looked for schoolyard fame, but now those days are long gone past
He's learned his lesson: don't go trusting everything they say
You get brightness from clear thinking - and not any other way.

Comment author: conchis 19 March 2009 12:09:31PM *  11 points [-]

Yvain, I enjoy your posts, and generally find them useful, informative, and well written.

I also recognize that this view is controversial in some circles, but one thing that would make me enjoy them rather more is if you managed to ferret out the implicit assumption that crops up every now and then that your rationalist protagonists are necessarily male. (Or at least predominantly so, I haven't been back to do an exhaustive stock-take of your gender specific pronoun usage, but I do recall being struck by this at least once before, so I figured it was worth a comment this time.)

Just to clarify, I don't mean Theo here. If you want to use a specifically male example, that's fine. But phrases like "the most important reason to argue with someone is to change his mind" and "[e]ither a person has enough of the rationalist virtues to overcome it, or he doesn't" strike me as problematic.

I'm not for a moment suggesting that you're being consciously sexist here. In fitting with the theme of this post, I spent a fair while rejecting others' calls for gender neutral language under the mistaken (largely emotional) impression that agreeing with them would have be an admission of some deep moral flaw in me, rather than merely a small and relatively painless step towards inclusiveness - and ultimately better communication.

Comment author: Yvain 19 March 2009 01:57:34PM *  13 points [-]

I'm glad you brought that up.

I've thought about this a few times, and I agree with you that it promotes sexism and is bad, but I just really hate using the phrases "he or she" every time I have to use a pronoun. A sentence like "A rationalist should ensure he or she justifies his or her opinion to himself or herself" is just too awkward to understand. And I am too much of a grammar purist to use "them" as a singular.

I used to use the gender-neutral pronoun "ze", but people told me they didn't understand it or didn't like it or thought it sounded stupid. And I tried using "she" as the default for a while, but people kept getting confused because they weren't expecting it, and trying to figure out where I'd mentioned a female.

I'm willing to accept whatever the common consensus is here. Maybe Less Wrong-ers are open-minded enough to accept "ze" where the average reader isn't.

(I've heard some people here use "ve" a few times, but from the context I gathered it was more of a way to refer to aliens/AIs/transhumans than a normal gender-neutral pronoun. Is this true?)

In response to Rationalist Fiction
Comment author: Yvain 19 March 2009 11:05:52AM *  8 points [-]

The Prince of Nothing. The Prince of Nothing. The Prince of Nothing. I'll say it as many times as I have to to get people on this blog to read it. The Prince of Nothing Trilogy. The Darkness That Comes Before. R. Scott Bakker.

True, Anasurimbor Kellhus is one of the "mutants", even to the point of the author explicitly stating the Dunyain spent a few centuries running a eugenics program to get an intellect of that stature. But the later books of the trilogy also go into some detail about the rationalist training Kellhus undergoes with the Dunyain, the methods he uses, and even a little of the social structure of the Ishual monastery. He's one of the perspective characters, and we see him using his techniques; there's always a strong sense of "I could do that", which remains right up until I actually try. And there's no better work to demonstrate the "sense that more is possible", or the ways in which a real rationalist would be the polar opposite of the "Spock" prototype, or a bunch of other things (disclaimer: many don't become fully clear until The Thousandfold Thought, the last book in the trilogy).

The Dunyain conception of rationality isn't exactly like our own, and rereading it recently there were a few things that bothered me, but overall it's basically the story of a fantasy hero who is as good at probability theory as Aragorn is at swordfighting, with similar results.

Comment author: komponisto 19 March 2009 07:17:53AM 6 points [-]

Sometimes the harsh approach has surprisingly good results. Example.

But Christopher Hitchens always struck me as too black-and-white and just plain irritating

Tangential, I know, but this surprises me. Hitchens, with his literary background, strikes me as a very nuanced thinker, attuned to the various shades of gray. (For example, he's by no means unmoved by religion's contributions to art and culture.) Maybe you're thinking of his talent for devastating rhetorical flourish, as in his infamous comments on Jerry Falwell?

Comment author: Yvain 19 March 2009 10:50:00AM 6 points [-]

I accept your correction. I've only seen Hitchens on TV a few times and never read his book. My introduction to him was in fact his Jerry Falwell comments. If you say he's more nuanced in his writings, I believe you.

Comment author: John_Maxwell_IV 18 March 2009 10:52:55PM 9 points [-]

The meter is inconsistent...

Comment author: Yvain 19 March 2009 01:33:47AM 4 points [-]

Oh, I am so happy to meet someone else who cares about/notices this sort of thing. And on a rationalist website, no less!

Despite my conservative tastes, I give Darmani a lot of credit for writing a rationalist's children poem. That clearly took a lot of time and energy.

How to Not Lose an Argument

109 Post author: Yvain 19 March 2009 01:07AM

Related to: Leave a Line of Retreat

Followup to: Talking Snakes: A Cautionary Tale, The Skeptic's Trilemma

"I argue very well. Ask any of my remaining friends. I can win an argument on any topic, against any opponent. People know this, and steer clear of me at parties. Often, as a sign of their great respect, they don't even invite me."

        --Dave Barry

The science of winning arguments is called Rhetoric, and it is one of the Dark Arts. Its study is forbidden to rationalists, and its tomes and treatises are kept under lock and key in a particularly dark corner of the Miskatonic University library. More than this it is not lawful to speak.

But I do want to talk about a very closely related skill: not losing arguments.

Rationalists probably find themselves in more arguments than the average person. And if we're doing it right, the truth is hopefully on our side and the argument is ours to lose. And far too often, we do lose arguments, even when we're right. Sometimes it's because of biases or inferential distances or other things that can't be helped. But all too often it's because we're shooting ourselves in the foot.

How does one avoid shooting one's self in the foot? In rationalist language, the technique is called Leaving a Social Line of Retreat. In normal language, it's called being nice.

First, what does it mean to win or lose an argument? There is an unspoken belief in some quarters that the point of an argument is to gain social status by utterly demolishing your opponent's position, thus proving yourself the better thinker. That can be fun sometimes, and if it's really all you want, go for it.

But the most important reason to argue with someone is to change his mind. If you want a world without fundamentalist religion, you're never going to get there just by making cutting and incisive critiques of fundamentalism that all your friends agree sound really smart. You've got to deconvert some actual fundamentalists. In the absence of changing someone's mind, you can at least get them to see your point of view. Getting fundamentalists to understand the real reasons people find atheism attractive is a nice consolation prize.

I make the anecdotal observation that a lot of smart people are very good at winning arguments in the first sense, and very bad at winning arguments in the second sense. Does that correspond to your experience?

Back in 2008, Eliezer described how to Leave a Line of Retreat. If you believe morality is impossible without God, you have a strong disincentive to become an atheist. Even after you've realized which way the evidence points, you'll activate every possible defense mechanism for your religious beliefs. If all the defense mechanisms fail, you'll take God on utter faith or just believe in belief, rather than surrender to the unbearable position of an immoral universe.

The correct procedure for dealing with such a person, Eliezer suggests, isn't to show them yet another reason why God doesn't exist. They'll just reject it along with all the others. The correct procedure is to convince them, on a gut level, that morality is possible even in a godless universe. When disbelief in God is no longer so terrifying, people won't fight it quite so hard and may even deconvert themselves.

But there's another line of retreat to worry about, one I experienced firsthand in a very strange way. I had a dream once where God came down to Earth; I can't remember exactly why. In the borderlands between waking and sleep, I remember thinking: I feel like a total moron. Here I am, someone who goes to atheist groups and posts on atheist blogs and has told all his friends they should be atheists and so on, and now it turns out God exists. All of my religious friends whom I won all those arguments against are going to be secretly looking at me, trying as hard as they can to be nice and understanding, but secretly laughing about how I got my comeuppance. I can never show my face in public again. Wouldn't you feel the same?

And then I woke up, and shook it off. I am an aspiring rationalist: if God existed, I would desire to believe that God existed. But I realized at that point the importance of the social line of retreat. The psychological resistance I felt to admitting God's existence, even after having seen Him descend to Earth, was immense. And, I realized, it was exactly the amount of resistance that every vocally religious person must experience towards God's non-existence.

There's not much we can do about this sort of high-grade long-term resistance. Either a person has enough of the rationalist virtues to overcome it, or he doesn't. But there is a less ingrained, more immediate form of social resistance generated with every heated discussion.

Let's say you approach a theist (let's call him Theo) and say "How can you, a grown man, still believe in something stupid like talking snakes and magic sky kings? Don't you know you people are responsible for the Crusades and the Thirty Years' War and the Spanish Inquisition? You should be ashamed of yourself!"

This suggests the following dichotomy in Theo's mind: EITHER God exists, OR I am an idiot who believes in stupid childish  things and am in some way partly responsible for millions of deaths and I should have lower status and this arrogant person who's just accosted me and whom I already hate should have higher status at my expense.

Unless Theo has attained a level of rationality far beyond any of us, guess which side of that dichotomy he's going to choose? In fact, guess which side of that dichotomy he's now going to support with renewed vigor, even if he was only a lukewarm theist before? His social line of retreat has been completely closed off, and it's your fault.

Here the two definitions of "winning an argument" I suggested before come into conflict. If your goal is to absolutely demolish the other person's position, to make him feel awful and worthless - then you are also very unlikely to change his mind or win his understanding. And because our culture of debates and mock trials and real trials and flaming people on Usenet encourages the first type of "winning an argument", there's precious little genuine mind-changing going on.

Really adjusting to the second type of argument, where you try to convince people, takes a lot more than just not insulting people outright1. You've got to completely rethink your entire strategy. For example, anyone used to the Standard Debates may already have a cached pattern of how they work. Activate the whole Standard Debate concept, and you activate a whole bunch of related thoughts like Atheists As The Enemy, Defending The Faith, and even in some cases (I've seen it happen) persecution of Christians by atheists in Communist Russia. To such a person, ceding an inch of ground in a Standard Debate may well be equivalent to saying all the Christians martyred by the Communists died in vain, or something similarly dreadful.

So try to show you're not just starting Standard Debate #4457. I remember once, during the middle of a discussion with a Christian, when I admitted I really didn't like Christopher Hitchens. Richard Dawkins, brilliant. Daniel Dennett, brilliant. But Christopher Hitchens always struck me as too black-and-white and just plain irritating. This one little revelation completely changed the entire tone of the conversation. I was no longer Angry Nonbeliever #116. I was no longer the living incarnation of All Things Atheist. I was just a person who happened to have a whole bunch of atheist ideas, along with a couple of ideas that weren't typical of atheists. I got the same sort of response by admitting I loved religious music. All of a sudden my friend was falling over himself to mention some scientific theory he found especially elegant in order to reciprocate2. I didn't end up deconverting him on the spot, but think he left with a much better appreciation of my position.

All of these techniques fall dangerously close to the Dark Arts, so let me be clear: I'm not suggesting you misrepresent yourself just to win arguments. I don't think misrepresenting yourself would even work; evolutionary psychology tells us humans are notoriously bad liars. Don't fake an appreciation for the other person's point of view, actually develop an appreciation for the other person's point of view. Realize that your points probably seem as absurd to others as their points seem to you. Understand that many false beliefs don't come from simple lying or stupidity, but from complex mixtures of truth and falsehood filtered by complex cognitive biases. Don't stop believing that you are right and they are wrong, unless the evidence points that way. But leave it at them being wrong, not them being wrong and stupid and evil.

I think most people intuitively understand this. But considering how many smart people I see shooting their own foot off when they're trying to convince someone3, some of them clearly need a reminder.

 

Footnotes

1: An excellent collection of the deeper and most subtle forms of this practice of this sort can be found in Dale Carnegie's How to Win Friends and Influence People, one of the only self-help books I've read that was truly useful and not a regurgitation of cliches and applause lights. Carnegie's thesis is basically that being nice is the most powerful of the Dark Arts, and that a master of the Art of Niceness can use it to take over the world. It works better than you'd think.

2: The following technique is definitely one of the Dark Arts, but I mention it because it reveals a lot about the way we think: when engaged in a really heated, angry debate, one where the insults are flying, suddenly stop and admit the other person is one hundred percent right and you're sorry for not realizing it earlier. Do it properly, and the other person will be flabbergasted, and feel deeply guilty at all the names and bad feelings they piled on top of you. Not only will you ruin their whole day, but for the rest of time, this person will secretly feel indebted to you, and you will be able to play with their mind in all sorts of little ways.

3: Libertarians, you have a particular problem with this. If I wanted to know why I'm a Stalin-worshipper who has betrayed the Founding Fathers for personal gain and is controlled by his base emotions and wants to dominate others by force to hide his own worthlessness et cetera, I'd ask Ann Coulter. You're better than that. Come on. And then you wonder why people never vote for you.

Comment author: thomblake 18 March 2009 08:54:34PM *  1 point [-]

once I get working survey software, though.

I work for a company that programs and hosts market research surveys - feel free to drop me a line and I'll see if we're interested.

Comment author: Yvain 18 March 2009 09:03:33PM *  1 point [-]

I was thinking more along the lines of something very simple and very free, since I have minimal computer skills and financial resources. I have SPSS for analysis, so all I need is a form that collects responses and sticks them into a spreadsheet. One of my friends helped me do a survey before, and I was planning to pester him until he showed me how to set something up.

If you're a professional, though, I defer to you if you want to do the Unofficial Less Wrong Survey instead. Or if you want to cooperate on it, email me at yvain314@hotmail.com. If not, no worries; I'm sure I can set something up eventually.

Comment author: JamesAndrix 18 March 2009 03:46:01PM 25 points [-]

Whenever a decision seems very close, I flip a coin to decide, and then I check to see if I wish the coin had gone the other way. If so then I go against the coin.

I might sometimes be 'agreeing' with the coin because I'm primed by its outcome, but overall I find it useful for saving time. I rarely regret decisions made this way.

Comment author: Yvain 18 March 2009 07:20:48PM *  31 points [-]

"Whenever you're called on to make up your mind,
and you're hampered by not having any,
the best way to solve the dilemma, you'll find,
is simply by spinning a penny.
No -- not so that chance shall decide the affair
while you're passively standing there moping;
but the moment the penny is up in the air,
you suddenly know what you're hoping."

That's from Piet Hein, a poet (and mathematician) I recommend. But I'm afraid I don't see the link to priming.

Comment author: prase 18 March 2009 12:10:56PM 2 points [-]

Can priming influence also longer held beliefs? I would suppose that since people usually dislike to change their mind, it becomes extremely important where they learn first about particular facts. For example it would be probably beneficial if children were first informed about evolution during some school excursion in a hi-tech laboratory, as opposed to during church sermon (even if the information were the same).

Comment author: Yvain 18 March 2009 07:16:33PM *  5 points [-]

I am not a psychologist, but every time I've come across priming it has been presented as a short-term phenomenon caused by temporary brain activation. I don't think what you're describing could be called priming.

But I have heard of some results along those lines. For example, if you learn something in for example a classroom, you are more likely to remember it on a test given in a classroom than in (for example) a church, and vice versa (link to study). And this even generalizes to people who learn something in a red room recalling it better in other red places (link to study) That makes effects like the one you describe sound possible, though I don't know of any studies of it directly.

Comment author: Matt_Simpson 18 March 2009 04:47:06PM *  4 points [-]

Can we use this to our advantage? Maybe tattoo "Less Wrong" on one's palm and look at it when in distress?

btw, this is almost the sort of thing a chaos magician would do. They would create a symbol by smashing the letters in Less Wrong together and then tattoo the created symbol on their palm. Or at least keep copies of it around all the time.. Perhaps the symbol has more priming power than mere words. Any evidence of this?

Comment author: Yvain 18 March 2009 07:06:22PM 1 point [-]

Not that I know of.

Comment author: ciphergoth 18 March 2009 01:19:17PM 3 points [-]

There even seems to be a tacit awareness that it's not actually magic, though I don't think the awareness ever becomes explicit.

AFAICT most chaos magicians believe in the supernatural, but at least some have been thoroughgoing materialists who believe only in the psychological power of suspension of disbelief.

Comment author: Yvain 18 March 2009 02:31:05PM *  5 points [-]

The few times I experimented with similar techniques (usually expressed as self-help rather than magic), I found it impossible to suspend disbelief, and ended up laughing at myself and giving up. I wonder if other rationalists would have the same experience, and if this would be different between self-selected rationalists and random people put through a course in a rationality dojo. I'd also like to see to whether this is mediated by the hypnotizability trait.

So many studies to do, so little status as a real psychologist who can do studies and stuff. I do want to run a survey of Less Wrong members and gather some demographic/other interesting data once I get working survey software, though.

Comment author: RobinHanson 18 March 2009 12:41:21PM 4 points [-]

Let me us distinguish "truth-seekers", people who respect and want truth, from "rationalists", people who personally know how to believe truth. We can build better institutions that produce truth if only we have enough support from truth-seekers; we don't actually need many rationalists. And having rationalists without good institutions may not produce much more shared accessible truth.

Comment author: Yvain 18 March 2009 02:22:40PM *  15 points [-]

I'm not sure I can let you make that distinction without some more justification.

Most people think they're truth-seekers and honestly claim to be truth-seekers. But the very existence of biases shows that thinking you're a truth-seeker doesn't make it so. Ask a hundred doctors, and they'll all (without consciously lying!) say they're looking for the truth about what really will help or hurt their patients. But give them your spiel about the flaws in the health system, and in the course of what they consider seeking the truth, they'll dismiss your objections in a way you consider unfair. Build an institution that confirms your results, and they'll dismiss the institution as biased or flawed or "silly". These doctors are not liars or enemies of truth or anything. They're normal people whose search for the truth is being hijacked in ways they can't control.

The solution: turn them into rationalists. They don't have to be black belt rationalists who can derive Bayes' Theorem in their sleep, but they have to be rationalist enough that their natural good intentions towards truth-seeking correspond to actual truth-seeking and allow you to build your institutions without interference.

Comment author: CannibalSmith 18 March 2009 08:53:05AM *  10 points [-]

Can we use this to our advantage? Maybe tattoo "Less Wrong" on one's palm and look at it when in distress?

Comment author: Yvain 18 March 2009 09:50:47AM *  17 points [-]

That's a very interesting idea. Again, it reminds me of certain forms of magic - for example, of magic practioners who used to surround themselves with imagery of the God of Music before a music recital on the theory it would help them play better. I wish there were studies as to the effectiveness of that, but it's kind of hard to get funding to study magic. I suspect that saying a prayer to God for help resisting temptation may work in the same way - you're activating your concept of God and with it the entire religious system of morality and self-control. In the same way, giving yourself a reminder of rationalism or even reciting one of those rationalist litanies might make the stuff you learn here more salient.

Tell you what - tattoo "Less Wrong" on your palm and post a picture on here, and I'll upvote ten of your comments. Ten karma points! Think about it! That would totally be worth it!

Comment author: Andrew 18 March 2009 07:12:31AM 2 points [-]

There's a relatively new branch of psychology called Terror Management Theory that specializes the general phenomena cited above for instances where one is reminded of one's mortality. I've only read a couple journal articles in the field, and I'm not a psychologist, but one experimental design in particular struck a cord in me.

I'm no longer certain of the study, but they primed their candidates with a short story about either life insurance policies or (for the control group) the imports and exports of a certain country. Then, they had the subjects try to complete a list of partially-spelled words specifically chosen so that they had two interpretations -- one morbid, the other benign. The only pair I remember was skull and skill, both derivable from S**LL.

Then, to get the cross-cultural study, they found these word pairs in modern Hebrew! How cool is that! I should dig that study up again. I wonder if they used the truth/death similarity exploited by the story of the Golem.

Comment author: Yvain 18 March 2009 08:11:30AM *  2 points [-]

Sheldon Solomon, one of the big names behind terror management theory, was the one who conducted the WTC study. He also did a related experiment in the same study where he made people think about their own deaths and found they were more likely to vote Bush afterwards. I think there's a description at the same link. Good catch.

Comment author: Vladimir_Nesov 18 March 2009 01:04:55AM 12 points [-]

Why do you expect the decision made in a default state to be more accurate than if it's modified by an unknown factor? If you don't expect to be primed in any particular direction, you may as well be swayed towards the right choice. Priming occurs even when you think of something yourself, so the advice could as well be extended to suggesting not to think about anything, especially about things relevant to the decision at hand, since those things are likely to influence the decision most. When any shadow can influence the outcome, your decision is probably no good to start with, and adding some unexpected noise won't hurt that much.

Comment author: Yvain 18 March 2009 01:16:59AM 12 points [-]

I'm interpreting priming as like random noise, since we don't know which direction it will move my estimate. Assuming my rational decision process is better than picking a result at random, adding random noise to it will on average lower its effectiveness.

See also: footnote 3. Since the people not primed by being in a school came to a different result, and since it seems they must have thought about schools at least once when deciding about a school funding initiative, it seems likely that there's something about an external stimulus that's different from an internal one.

Never Leave Your Room

66 Post author: Yvain 18 March 2009 12:30AM

Related to: Priming and Contamination

Psychologists define "priming" as the ability of a stimulus to activate the brain in such a way as to affect responses to later stimuli. If that doesn't sound sufficiently ominous, feel free to re-word it as "any random thing that happens to you can hijack your judgment and personality for the next few minutes."

For example, let's say you walk into a room and notice a briefcase in the corner. Your brain is now the proud owner of the activated concept "briefcase". It is "primed" to think about briefcases, and by extension about offices, business, competition, and ambition. For the next few minutes, you will shift ever so slightly towards perceiving all social interactions as competitive, and towards behaving competitively yourself. These slight shifts will be large enough to be measured by, for example, how much money you offer during the Ultimatum Game. If that sounds too much like some sort of weird New Age sympathetic magic to believe, all I can say is Kay, Wheeler, Bargh, and Ross, 2004.1

We've been discussing the costs and benefits of Santa Claus recently. Well, here's one benefit: show Dutch children an image of St. Nicholas' hat, and they'll be more likely to share candy with others. Why? The researchers hypothesize that the hat activates the concept of St. Nicholas, and St. Nicholas activates an idealized concept of sharing and giving. The child is now primed to view sharing positively. Of course, the same effect can be used for evil. In the same study, kids shown the Toys 'R' Us logo refused to share their precious candy with anyone.

But this effect is limited to a few psych laboratories, right? It hasn't done anything like, you know, determine the outcome of a bunch of major elections?



I am aware of two good studies on the effect of priming in politics. In the first, subjects were subliminally2 primed with either alphanumeric combinations that recalled the 9/11 WTC attacks (ie "911" or "WTC"), or random alphanumeric combinations. Then they were asked to rate the Bush administration's policies. Those who saw the random strings rated Bush at an unenthusiastic 42% (2.1/5). Those who were primed to be thinking about the War on Terror gave him an astounding 75% (3.75/5). This dramatic a change, even though none of them could consciously recall seeing terrorism-related stimuli.

In the second study, scientists analyzed data from the 2000 election in Arizona, and found that polling location had a moderate effect on voting results. That is, people who voted in a school were more likely to support education-friendly policies, people who voted in a church were more likely to support socially conservative policies, et cetera. The effect seems to have shifted results by about three percentage points. Think about all the elections that were won or lost by less than three percent...

Objection: correlation is not causation! Religious people probably live closer to churches, and are more likely to know where their local church is, and so on. So the scientists performed an impressive battery of regression analyses and adjustments on their data. Same response.

Objection: maybe their adjustments weren't good enough! The same scientists then called voters into their laboratory, showed them pictures of buildings, and asked them to cast a mock vote on the education initiatives. Voters who saw pictures of schools were more likely to vote yes on the pro-education initiatives than voters who saw control buildings.

What techniques do these studies suggest for rationalists? I'm tempted to say the optimal technique is to never leave your room, but there are still a few less extreme things you can do. First, avoid exposure to any salient stimuli in the few minutes before making an important decision. Everyone knows about the 9-11 terrorist attacks, but the War on Terror only hijacked the decision-making process when the subjects were exposed to the related stimuli directly before performing the rating task3.

Second, try to make decisions in a neutral environment and then stick to them. The easiest way to avoid having your vote hijacked by the location of your polling place is to decide how to vote while you're at home, and then stick to that decision unless you have some amazing revelation on your way to the voting booth. Instead of never leaving your room, you can make decisions in your room and then carry them out later in the stimulus-laden world.

I can't help but think of the long tradition of master rationalists "blanking their mind" to make an important decision. Jeffreyssai's brain "carefully put in idle" as he descends to a bare white room to stage his crisis of faith. Anasûrimbor Kellhus withdrawing into himself and entering a probability trance before he finds the Shortest Path. Your grandmother telling you to "sleep on it" before you make an important life choice.

Whether or not you try anything as formal as that, waiting a few minutes in a stimulus-free environment before a big decision might be a good idea.

 

Footnotes

1: I bet that sympathetic magic probably does have strong placebo-type effects for exactly these reasons, though.

2: Priming is one of the phenomena behind all the hype about subliminal advertising and other subliminal effects. The bad news is that it's real: a picture of popcorn flashed subliminally on a movie screen can make you think of popcorn. The good news is that it's not particularly dangerous: your thoughts of popcorn aren't any stronger or any different than they'd be if you just saw a normal picture of popcorn.

3: The obvious objection is that if you're evaluating George Bush, it would be very strange if you didn't think of the 9-11 terror attacks yourself in the course of the evaluation. I haven't seen any research addressing this possibility, but maybe hearing an external reference to it outside the context of your own thought processes is a stronger activation than the one you would get by coming up with the idea yourself.

Comment author: anonym 17 March 2009 11:47:22PM *  16 points [-]

Leibniz, Da Vinci, Pascal, Descartes, and John von Neumann spring immediately to mind for me.

There's also Poincaré, often considered the last universalist. Kant is famous as a philosopher, but also worked in astronomy. Bertrand Russell did work in philosophy as well as mathematics, and was something of a generalist. Noam Chomsky is the linguist of the 20th century, and if you consider any of his political and media analysis outside of linguistics to be worthwhile, he's another. Bucky Fuller. Charles Peirce. William James. Aristotle. Goethe. Thomas Jefferson. Benjamin Franklin. Omar Khayyám.


Just thought of Gauss, who in addition to his work in mathematics did considerable work in physics.

Herbert Simon: psychology and computer science (got an economics Nobel).

Alan Turing: don't know how I could have forgotten him.

Norbert Wiener.

In response to comment by anonym on Rational Me or We?
Comment author: Yvain 17 March 2009 11:57:48PM 8 points [-]

Good answers. Also, Pierre-Simon Laplace, one of the inventors of Bayesian statistics, was also an excellent astronomer and physicist (and briefly the French Minister of the Interior, of all things)

In response to Rational Me or We?
Comment author: mtraven 17 March 2009 08:00:20PM 6 points [-]

For whatever reason, the community here (so-called "rationalists") is heavily influenced by overly-individualistic ideologies (libertarianism, or in its more extreme forms, objectivism). This leads to ignoring entire realms of human phenomena (social cognition) and the people who have studied them (Vygotsky, sociologists of science, ethnomethodology). It's not that social approaches to cognition provide a magic bullet -- they just provide a very different perspective on how minds work. Imagine if you stop believing that beliefs are in the head and locate themselves in a community or institution. If interested, you could start with How Institutions Think by Mary Douglas.

In response to comment by mtraven on Rational Me or We?
Comment author: Yvain 17 March 2009 11:52:35PM *  14 points [-]

I am guilty as charged in being much more familiar with individualistic than socially oriented ideologies.

Why don't you write some posts about techniques or discoveries from socially-oriented science that could help rationalists?

In response to comment by Yvain on Science vs. art
Comment author: PhilGoetz 17 March 2009 02:40:07PM 5 points [-]

Sure; but then why does it matter whether you yourself make the literature? Shouldn't you be even more fulfilled by becoming an investment banker, and giving money to support a dozen novelists?

Yet I don't know any novelists who would enjoy that.

In response to comment by PhilGoetz on Science vs. art
Comment author: Yvain 17 March 2009 07:33:10PM 4 points [-]

That's cause there's a difference between reason and emotions?

I would feel a greater sense of fulfillment giving a kid an ice cream cone and seeing his eyes light up, than I would donating $1,000 to a a children's charity and knowing it would help dozens of children in need. I'm not saying that's better, just that my pre-programmed emotional responses react more strongly to it.

Your question seems to assume becoming a novelist is fundamentally a charitable act, or that novelists are working to maximize total world utility (unlike...everyone else?). I think most of them are just people who really, really like writing. I remember one novelist who said that the only valid excuse for becoming a writer was that you can't keep yourself from writing even if you try.

Even rejecting all that, the world does need a certain number of novelists, and not every novelist could succeed as an investment banker. If you assess yourself as having a certain high amount of literary skill and no even higher amounts of skill in other areas, then it may be that your best bet for contributing value to the world is in writing. Shakespeare's probably was.

Comment author: taw 17 March 2009 12:41:40PM 0 points [-]

A software request. Can we get #-links to footnotes? I wanted to tweet footnote 2, but it doesn't have any anchor. Or put it into a separate post, as it's awesome.

Comment author: Yvain 17 March 2009 03:24:34PM *  2 points [-]

It was already possible on the software, I was just too lazy to do it. Added anchors footnote1 and footnote2.

Comment author: Kaj_Sotala 17 March 2009 12:26:03PM 6 points [-]

I find the Ern Malley episode a bit puzzling. Yes, if we're talking about, say the origin of a religious story, then it's possible to reveal the theory as false. But there isn't such a thing as "true" or "false" in prose. Even if McAuley and Stewart purposely wrote what they considered to be bad poems, what does their intent matter if they produced poems others thought were good? If I like a book, my liking of it doesn't become "wrong" or "mistaken" if it's revealed that the author was actually trying to write a bad book, anymore than me disliking a book becomes mistaken simply because the author was trying to write a good book.

Comment author: Yvain 17 March 2009 12:38:02PM *  13 points [-]

I think modernist poetry is invested in the belief that there is something special about excellent modernist poems; that is, their structure produces emotions or enlightenment not found in random series of words, and modernist poets deserve high status because they can create structures with this emotion or enlightenment.

A competing hypothesis is that the arrangement of words in modernist poetry has no particular value at all, and that people who claim it has value are only doing so to gain status within the modernist poet community.

I think the important part of the Malley experiment was that the two hoaxers created their poems from randomly chosen phrases taken by opening books to random pages. If there's no way to distinguish a random collection of words from a great modernist poem, then people who can create "great modernist poems" aren't special and don't deserve high status, and it supports the hypothesis that people who claim to have been moved by modernist poetry are faking it to look highbrow.

But in reference to your point, I remember reading Lovecraft's poem "Nathicana" and being very impressed by it. I was astonished to discover a few years later that he wrote it as a parody of people who stick too much emotion into their poetry. I was only slightly mollified to learn I wasn't the only person who liked it and that it was often held up as an example of how a deliberately bad poem can sometimes be pretty good.

In response to comment by bentarm on Science vs. art
Comment author: PhilGoetz 17 March 2009 03:39:24AM 1 point [-]

Yes, and I almost addressed that in the original post, but decided the explanation would distract. My impression is that there's plenty of room for great works; and people still read and listen to crap. The kicker is that if what you care about is people who read your book, then you probably value your book for its instrumental value. Yet all authors care about whether people read their books. Does this mean no authors place intrinsic value on their works? I think the answer is more complicated than that, and although its answer is probably at some point necessary to get further towards an answer to the question I'm asking in this post, it is much further down the road than what I'm asking here

In response to comment by PhilGoetz on Science vs. art
Comment author: Yvain 17 March 2009 10:31:32AM *  6 points [-]

"If what you care about is people who read your book, then you probably value your book for its instrumental value"

How much would you value the following situations?

  1. By odd coincidence, a brilliant work of literature forms spontaneously in the Andromeda galaxy. No one ever reads it.

  2. An author writes a brilliant work of literature, then immediately burns it and shoots himself, so that it is lost to all time.

  3. An author writes a brilliant work of literature, then buries it and never tells another living soul.

  4. An author writes a brilliant work of literature. He lets one other person read it, then buries it and never tells another living soul.

  5. A Chinese author writes a brilliant work of literature. Millions of Chinese people read and enjoy it, but is never translated into English and neither you nor anyone you know ever hear about it.

  6. An English author writes a brilliant work of literature. You and many others read and enjoy it.

Through situations like these, I've decided that what I value in art has something to do with people enjoying it. I place minimal value on 1-4, but high value on 5 and 6. I don't think I believe in the simple form of 'only this one thing called happiness is important' that is vulnerable to the wirehead trick, but I do think that everything valuable is linked to a form of what we would call happiness. I would say my terminal value is "enjoyment of great literature" rather than the simple existence of the literature. The same is probably true of science's terminal value: Bayes Theorem written on a rock in Andromeda has no value, but the clarity and knowledge and enjoyment of science that people who know the theorem get does.

The "Spot the Fakes" Test

48 Post author: Yvain 17 March 2009 12:52AM

Followup to: Are You a Solar Deity?

James McAuley and Harold Stewart were mid-20th century Australian poets, and they were not happy. After having society ignore their poetry in favor of "experimental" styles they considered fashionable nonsense, they wanted to show everyone what they already knew: the Australian literary world was full of empty poseurs.

They began by selecting random phrases from random books. Then they linked them together into something sort of like poetry. Then they invented the most fashionable possible story: Ern Malley, a loner working a thankless job as an insurance salesman, writing sad poetry in his spare time and hiding it away until his death at an early age. Posing as Malley's sister, who had recently discovered the hidden collection, they sent the works to Angry Penguins, one of Australia's top experimental poetry magazines.

You wouldn't be reading this if the magazine hadn't rushed a special issue to print in honor of "a poet in the same class as W.H. Auden or Dylan Thomas".

The hoax was later revealed1, everyone involved ended up with egg on their faces, and modernism in Australia received a serious blow. But as I am reminded every time I look through a modern poetry anthology, one Ern Malley every fifty years just isn't enough. I daydream about an alternate dimension where people are genuinely interested in keeping literary criticism honest. In this universe, any would-be literary critic would have to distinguish between ten poems generally recognized as brilliant that he'd never seen before, and ten pieces of nonsense invented on the spot by drunk college students, in order to keep his critic's license.

Can we refine this test? And could it help Max Muller with his solar deity problem?

In the Malley hoax, McAuley and Steward suspected that a certain school of modernist poetry was without value. Because its supporters were too biased to admit this directly, they submitted a control poem they knew was without value, and found the modernists couldn't tell the difference. This suggests a powerful technique for determining when something otherwise untestable might be, as Neal Stephenson calls it, bulshytte.

Perhaps Max Muller thinks Hercules is a solar deity. He will write up a argument for this proposition, and submit it for consideration before all the great mythologists of the world. Even if these mythologists want to be unbiased, they will have a difficult time of it: Muller has a prestigious reputation, and they may not have any set conception of what does and doesn't qualify as a solar deity.

What if, instead of submitting one argument, Muller submitted ten? One sincere argument for why Hercules is a solar deity, and other bogus arguments for why Perseus, Bellerophon, Theseus, et cetera are solar myths (which he has nevertheless constructs to the best of his ability). Then he instructs the mythologists "Please independently determine which of these arguments is true, and which ones I have just come up with by writing 'X is a solar deity' as my bottom line and then inventing fake justifications for the fact?" If every mythologist finds the Hercules argument most convincing, then that doesn't prove anything about Hercules but it at least shows Muller has a strong case. On the other hand, if they're all convinced by different arguments, or find none of the arguments convincing, or worst of all they all settle on Bellerophon, then Dr. Muller knows his beliefs about Hercules are quite probably wishful thinking.

This method hinges on Dr. Muller's personal honesty: a dishonest man could simply do a bad job arguing for Theseus and Bellerophon. What if we thought Dr. Muller was dishonest? We might find another mythologist whom independent observers rate as equally persuasive as Dr. Muller, and ask her to come up with the bogus arguments.

The rationalists I know sometimes take a dim view of the humanities as academic disciplines. Part of the problem is the seeming untestability of their conclusions through good, blinded experimental methods. I don't think most humanities professors are really looking all that hard for such methods. But for those who are, I consider this technique a little better than nothing2.

Footnotes

1: The Sokal Affair is another related hoax. Wikipedia's Sokal Hoax page has some other excellent examples of this sort of test.

2: One more example where this method could prove useful. I remember debating a very smart Christian on the subject of Biblical atrocities. You know, stuff about death by stoning for minor crimes, or God ordering the Israelites to murder women and enslave children - that sort of thing. My friend, who was quite smart, was always able to come up with a superficially plausible excuse, and it was getting on my nerves. But having just read Your Strength as a Rationalist, I knew that being able to explain anything wasn't always a virtue. I proposed the following experiment: I'd give my friend ten atrocities commanded by random Bronze Age kings generally agreed by historical consensus to be jerks, and ten commanded by God in the Bible. His job would be to determine which ten, for whatever reason, really weren't all that bad. If he identified the ten Bible passages, that would be strong evidence that Biblical commandments only seemed atrocious when misunderstood. But if he couldn't tell the difference between God and Ashurbanipal, that would prove God wasn't really that great. To my disgust, my friend knew his Bible so well that I couldn't find any atrocities he wasn't already familiar with. So much for that technique. I offer it to anyone who debates theists with less comprehensive knowledge of Scripture.

Comment author: TreeFrog 16 March 2009 03:40:31AM 0 points [-]

Word of warning: I have had a couple glasses of Firefly vodka mixed with lemonade. The everloving devil's brew, I tell you.

Yvain is most likely smarter than me and has the additional bonus of caring intensely about subjects I dabble in. However, he always delivers on the entertainment. That eardrop thing was ludicrously fun.

it seems to me that "a general strategy for approaching this sort of problem" has the same pitfalls as always trusting conventional wisdom over contrarian wisdom.

Comment author: Yvain 16 March 2009 08:48:14PM 1 point [-]

I'm glad you find my articles entertaining, but I don't see how you can say a general strategy has certain pitfalls without knowing what the strategy is. Or do you think just trying to approach this problem from a rationalist perspective is a mistake?

Comment author: Johnicholas 15 March 2009 10:00:41PM 3 points [-]

Isn't this the same problem as overfitting? I think there are techniques against overfitting, even a contest, the Performance Prediction Challenge

Comment author: Yvain 16 March 2009 08:45:58PM 2 points [-]

My grasp of statistics is weak, but I am missing the connection to overfitting. I always thought overfitting to mean something like interpreting a series of coin flips producing HHTHT to mean the coin was fixed to produce HHTHT. Two characteristics of overfitting, then, are that you can only overfit the same theory to one piece of data, and that getting more data will ruin the overfitting.

But here, you can apply the "sun god" concept to practically any myth you want (if you're a good enough arguer). And I imagine that if someone discovered new information about Hercules, Max Muller would be able to think up a reason why it supported the sun god theory.

Comment author: James_Miller 16 March 2009 12:05:03AM 6 points [-]

I published two academic articles showing the relationship between game theory and Greek Mythology. A somewhat objective test of the validity of this relationship is whether Greek mythology can be used to help students achieve a better (as measured by tests) understanding of game theory.

Perhaps a somewhat useful measure of the relationship between two languages is how much learning one helps you master the other.

Comment author: Yvain 16 March 2009 08:36:29PM *  3 points [-]

If it's not a copyright issue/too much trouble, I'd like to see those articles. I've messaged you my email address.

Comment author: jimrandomh 15 March 2009 10:15:29PM 6 points [-]

Your criticism of comparative linguistics is unfair, because the goal of comparative linguistics is not to determine whether two languages are related, but to measure a degree of relatedness. Comparative linguists try to find a set of transformations (from a list of transformation templates which have been observed to occur) which go from one language's phonology to the other's, where the size of that set is inversely proportional to relatedness, and a percentage of words in common. From these measures, linguists can make falsifiable predictions about how long societies have been isolated from each other, when works were written, and so on.

Comment author: Yvain 15 March 2009 10:39:34PM 3 points [-]

Okay. I don't know much about linguistics, so I accept your correction.

Would you call the people who are trying to determine whether Nostratic and Amerind are real families, or whether Basque is related to Caucasian languages, and those sorts of things "comparative linguists"? If not, what would you call them?

Comment author: Eliezer_Yudkowsky 15 March 2009 07:17:13PM 1 point [-]

If they're fictional scenarios, then you're matching the taste of the students (in fictional answers) against the taste of the teachers; that may work to propagate a school, but how do you keep the tastes real?

Comment author: Yvain 15 March 2009 10:02:57PM *  7 points [-]

Then use scenarios that actually happened. From history, business, people's personal lives, et cetera. For example: "Here is a brief description of the Byzantine Empire in 1200. The Emperor decided to change the tax policy in the following way. Predict what happened." Gives an unfair advantage to anyone who knows a lot of history (or in this case economics), but if you vary the cases enough and use little-known enough examples you might be able to control for that.

Another example: "Here's a psych profile of my friend John, and a psych profile of his girlfriend Sally. They started dating ten years ago. Predict what happened."

Comment author: Yvain 15 March 2009 07:42:07PM 2 points [-]

Here is a stupid one: Detective stories. Like Encyclopedia Brown, but subtler. And with false leads. I don't think normal mass-market detective stories would work, because they may try to deliberately choose an irrational answer to surprise you. But special ones written by rationalists for rationalists could be a fun distraction if nothing else.

Are You a Solar Deity?

34 Post author: Yvain 15 March 2009 07:30PM

Max Muller was one of the greatest religious scholars of the 19th century. Born in Germany, he became fascinated with Eastern religion, and moved to England to be closer to the center of Indian scholarship in Europe. There he mastered English and Sanskrit alike to come out with the first English translation of the Rig Veda, the holiest book of Hinduism.

One of Muller's most controversial projects was his attempt to interpret all pagan mythologies as linked to one another, deriving from a common ur-mythology and ultimately from the celestial cycle. His tools were exhaustive knowledge of the myths of all European cultures combined with a belief in the interpretive power of linguistics.

What the significance of Orpheus' descent into the underworld to reclaim his wife's soul? The sun sets beneath the Earth each evening, and returns with renewed brightness. Why does Apollo love Daphne? Daphne is cognate with Sanskrit Dahana, the maiden of the dawn. The death of Hercules? It occurs after he's completed twelve labors (cf. twelve signs of zodiac) when he's travelling west (like the sun), he is killed by Deianeira (compare Sanskrit dasya-nari, a demon of darkness) and his body is cremated (fire = the sun).  His followers extended the method to Jesus - who was clearly based on a lunar deity, since he spent three days dead and then returned to life, just as the new moon goes dark for three days and then reappears.

Muller's work was massively influential during his time, and many 19th century mythographers tried to critique his paradigm and poke holes in it. Some accused him of trying to destroy the mystery of religion, and others accused him of shoddy scholarship.

R.F. Littledale, an Anglican apologist, took a completely different route. He claimed that there was, in fact, no such person as Professor Max Muller, holder of the Taylorian Chair in Modern European Languages. All these stories about "Max Muller" were nothing but a thinly disguised solar myth.

Littledale begins his argument by noting Muller's heritage. He was supposedly born in Germany, only to travel to England when he came of age. This looks suspiciously like the classic Journey of the Sun, which is born in the east but travels to the west. Muller's origin in Germany is a clear reference to Germanus Apollo, one of the old appelations of the Greek sun god.

His Christian name must be related to Latin "maximus" or Sanskrit "maha", meaning great, a suitable description of the King of Gods, and his surname is cognate with Mjolnir, the mighty hammer of the sky god Thor. His claim to fame is bringing the ancient wisdom of the East to the people of the West - that is, illuminating them with eastern light.

Muller teaches at Oxford for the same reason that Genesis describes the sky as "the waters above" and the Egyptians gave Ra a solar barge: ancient people interpreted the sky as a river, and the sun as crossing that river upon his chariot (perhaps an ox-drawn chariot, fording the river?). His chair at Oxford is the throne of the sky, his status as Taylorian Professor because "he cuts away with his glittering shears the ragged edges of cloud; he allows the...cuttings from his workshop, to descend in fertilizing showers upon the earth."

I could go on; instead I recommend you read the original essay. The take-home lesson is that any technique powerful enough to prove that Hercules is a solar myth is also powerful enough to prove that anyone is a solar myth. Muller lacked the strength of a rationalist: the ability to be more confused by fiction than by reality. This makes the Hercules theory useless, but that is not immediately apparent on a first or even a second reading of Muller's work. When reading Muller's work, the primary impression one gets is "Wow, this man has gathered a lot of supporting evidence."

This is a problem encountered in many fields of scholarship, especially "comparative" anything. In comparative linguistics, for example, it's usually possible to make a case that two languages are related good enough to convince a layman, no matter which two languages or how distant they may be. In comparative religion, we get cases like this blog's recent discussion over the possible derivation of Esther and Mordechai defeating Haman from Ishtar and Marduk defeating Humbaba. The less said about comparative literature, the better, although I can't help but quote humor writer Dave Barry:

Suppose you are studying Moby-Dick. Anybody with any common sense would say that Moby-Dick is a big white whale, since the characters in the book refer to it as a big white whale roughly eleven thousand times. So in your paper, you say Moby-Dick is actually the Republic of Ireland. Your professor, who is sick to death of reading papers and never liked Moby-Dick anyway, will think you are enormously creative. If you can regularly come up with lunatic interpretations of simple stories, you should major in English.

The worst (but most fun to read!) are in pseudoscience, where plausible sounding comparisons can prove almost anything. Did you know the Mayans believed in a lost homeland called Atzlan, the Indonesians believed in a lost island called Atala, and the Greeks believed in a lost continent called Atlantis? Likewise, did you know that Nostradamus predicted a great battle involving Germany and "Hister", which sounds almost like "Hitler"?

Yet it would be a mistake to reject all such comparisons. In fact, I have thus far been enormously unfair to Professor Muller, whose work established several correspondences still viewed as valid today. Virtually all modern mythologists accept that the Hindu Varuna is the Greek Uranus, and that the Greek sky god Zeus equals the Hindu sky god Dyaus Pita and the Roman Jupiter (compare to Latin deus pater, meaning God the Father). Likewise, comparative linguists are quite certain that all modern European languages and Sanskrit derive from a common Indo-European root, and in my opinion even the Nostratic project - an ambitious attempt to link Semitic, Indo-European, Uralic. and a bunch of other languages - is at least worth consideration.

We need a test to distinguish between true and false correspondences. But the standard method, making and testing predictions, is useless here. A good mythologist already knows the stories of Varuna and Uranus. The chances of discovering a new fact that either confirms or overturns the Varuna-Uranus correspondence is not even worth considering.

Mark Rosenfelder has an excellent article on chance resemblances between languages which offers a semi-formal model for spotting dubious comparisons. But such precision may not be possible when comparing two deities.

I have what might be a general strategy for approaching this sort of problem, which I will present tomorrow. But how would you go about it?

Comment author: Swimmy 15 March 2009 05:28:19PM *  9 points [-]

So, applying to some popular claim, say the resurrection of Jesus. The argument is that, instead of concluding, "It's highly unlikely that Jesus resurrected; his disciples were either mistaken, lying, or hallucinating; let's talk about the things we can test instead," attempts to assess the probability of surviving crucifixion according to available evidence are superior?

I agree on some level, but many "ignore" decisions are based on perfectly defensible Bayesian priors. Not ones that apply 0% probability to events, but I believe I could successfully argue that each of the 4 above explanations for the resurrection are ALL more likely than supernatural resurrection itself. And because of my knowledge of cognitive bias, I think I could well argue that the dismissive claims are more likely than surviving crucifixion as well.

Am I missing the point?

Edit: Actually, I think I see the error I made already. Explaining "hallucinating, mistaken, or lying" in terms of cognitive bias is much, much better than any of those explanations on their own.

Comment author: Yvain 15 March 2009 05:57:12PM 10 points [-]

I agree, many "ignore" decisions are based on perfectly defensible priors. I think this technique is only useful in certain circumstances.

If you care enough about the Resurrection that you want to be sure you have it exactly right (and many atheists don't care that much, and I don't blame them) then you need to spend some time looking for alternate explanations like Richard Carrier's.

It would be nice if there were some way to confirm Carrier's hypothesis in the same way Marinatos confirmed his Atlantis hypothesis, but I can't think of one. So you may end with a probability of epsilon for miracle, 70% for hallucination/mistake/lie, and 30% for Carrier. You now know more about the Resurrection than you did before, even though "hallucination/mistake/lie" is still your best guess (and you will know even more if you can break hallucination/mistake/lie down into the actual motivations or mechanisms behind lying or hallucinating in that situation).

I think this especially matters in cases where you have some reason to doubt the hallucination/mistake/lie hypothesis. If you personally knew St. Peter, and you couldn't possibly imagine him ever lying about anything, then restricting yourself to the dichotomy "Peter is a liar" or "Jesus rose from the dead" will tie you into epistemic knots. Under those circumstances, you'd be really happy you'd taken the extra few minutes to find Carrier's theory.

One thing I didn't make clear in the article and which you've caught is that the clever explanation isn't always or even usually better than the simple explanation. Sometimes people are just liars or mistaken, and to always use the clever explanation is as dangerous or worse than always using the simple explanation.

Ultimately this technique is about not prematurely stopping your search. If someone says "I saw Jesus three days after he was crucified", there's an overwhelming urge to ignore him, since your brain may leap to equate that with the absurd claim that he rose from the dead. But if you can tease apart those claims, you have another alternative. That alternative may be wrong, and you may end up rejecting it, but if you're unhappy with the previous dichotomy you ought to at least make sure you know it's there.

Comment author: MichaelHoward 14 March 2009 03:27:03PM 2 points [-]

Yvain,

Do you have a blog or home page with more material you've written? Failing that, is there another site (apart from OB) with contributions from you that might be interesting to LW readers?

Comment author: Yvain 15 March 2009 04:09:35PM 2 points [-]

Thanks for your interest. My blog is of no interest to anyone but my immediate personal friends, but I am working on a website. I'll let you know when it's up.

In response to comment by Yvain on Closet survey #1
Comment author: scientism 15 March 2009 01:56:36PM 13 points [-]

I wouldn't personally object, no. This is happening every day and, like most people, I do nothing. The difference is I don't think I'm supposed to be doing anything either. That isn't to say we should live in a society without laws or moral strictures; you need a certain amount of protection for society to function at all. You can't condone random violence. But this is a pragmatic rather than altruistic concern.

In response to comment by scientism on Closet survey #1
Comment author: Yvain 15 March 2009 03:51:03PM *  13 points [-]

Hm. Upvoted for an honest answer and lack of dissembling. Let's make it harder.

You have a button. If you press the button, you will receive a (free!) delicious pie, and a random child will be tortured for one year. No one will ever know there was any connection to you, and you can even erase your memory so that you won't feel guilty about it afterwards. Assume you like pie. Do you press the button?

In response to comment by Yvain on Soulless morality
Comment author: billswift 15 March 2009 03:37:32PM *  2 points [-]

Sorry for the confusion. A person's primary value should be himself. His value to others is what he contributes. I don't expect you to value me, except for what I may contribute, through my work or writing or whatever, to whatever you value.

Comment author: Yvain 15 March 2009 03:47:02PM 3 points [-]

Same question as I asked scientism on the other thread: if I started torturing a randomly selected 8-year old child, would this bother you? Assume the child has never contributed anything important, and that my torture will stop before it kills him or scars him so badly that he can't contribute in the future.

Comment author: Roko 15 March 2009 01:23:05PM 3 points [-]

From Wikipedia: "The prevalence of Charles Bonnet Syndrome has been reported to be between 10% and 40%; a recent Australian study has found the prevalence to be 17.5% [2]. Two Asian studies, however, report a much lower prevalence [5].[6]. The high incidence of non-reporting of this disorder is the greatest hindrance to determining the exact prevalence; non-reporting is thought to be as a result of sufferers being afraid to discuss the symptoms out of fear that they will be labelled insane[4]."

  • Yvain, your posts are an incredible wake-up call as to the astounding weirdness of the everyday world, to an extent that I ought to have expected but didn't. Entire countries, scientists, governments and all, are irrationally frightened of "fan death"? I wouldn't have believed it.

~10% of people see elves?! But don't report it because they think they'll get institutionalized? I would not have believed that.

Comment author: Yvain 15 March 2009 01:40:25PM 6 points [-]

I admit that I remembered hearing about Charles Bonnet syndrome before, and linked to the Wikipedia page without reading it. 10 to 40 PERCENT?!

Looking at the literature, I find some evidence that the real prevalence rate is closer to 1%, and that the higher numbers are only true of people with impaired vision. That's still impressive, though.

Comment author: bentarm 15 March 2009 12:36:22PM *  18 points [-]

I agree with most of this, but I think you're skipping a really important issue:

"There's a big difference between dismissing that whole Lost Continent of Atlantis story, and prematurely dismissing it."

Well, sure, but we need some way of deciding when our dismissal is premature. I mentioned this in the comments on Talking Snakes as well... there is certainly some room for the absurdity heuristic - my time is valuable, I can't evaluate the evidence for every crank claim in the world (I know several academics who could easily spend their entire lives checking "proofs" that P=NP if they did this). I have to reject some of them out of hand - the issue is: which ones?

If someone tells me they've built a perpetual motion machine in the back garden from tin cans and elastic bands, I'm not going to waste even 10 minutes of my life trying to replicate it. If Steven Hawking tells me he's built one, I'll at least give the matter some consideration. The real issue is where we draw the line between claims which are too absurd to bother looking at the evidence for and those we should take the time out to evaluate, you don't seem to have said anything yet that will help us to make that decision (perhaps you're getting there).

Comment author: Yvain 15 March 2009 01:13:25PM *  4 points [-]

Good points.

I don't have such a formula for drawing the line between "too absurd" and "go and evaluate" (I will be mentioning one technique in a while that might help a little, but it's a complicated technique and there's no reason to do it unless you've already decided something's important).

My point is that we should increase the amount of potential value we expect from investigating seemingly stupid claims for two reasons. One, the seemingly stupid claims may be true (ie evolution). Two, the seemingly stupid claims may be false, but there may be some hidden reason for their existence that is still a worthwhile discovery (ie religion leads to evo psych, Atlantis leads to Santorini).

But that's all I'm saying - increase the potential value you expect from investigating. In the case of the perpetual motion inventor, the case against perpetual motion isn't the absurdity heuristic, it's a physical law that's been proven about as well as anything in science - so no need to increase the value of investigation on those grounds.

You should consider the possibility that you'll discover something interesting about the human mind when trying to figure out why people keep inventing perpetual motion machines, but if you're a physicist who's very busy and doesn't have much interest in wild goose chases after some psychological phenomenon, then it's completely fair to leave that to the psychologists and tell the perpetual motion inventor to take a hike.

Likewise, you should assign nonzero probability to the possibility that the perpetual motion machine inventor has discovered something interesting that's not a perpetual motion machine - I think some people used to call the solar sail a perpetual motion machine, and even though they were wrong the solar sail is still quite an interesting discovery. This is worth consideration if the inventor is someone smart enough that you don't expect them to make elementary mistakes (ie Stephen Hawking).

If it was just some random guy, I would join you in not paying him any attention, unless I was feeling especially interested in cognitive biases relating to perpetual motion machine construction that day.

Summary: Correct for certain possibilities when deciding whether or not to reject something out of hand, but if you still want to reject something out of hand after making the corrections, do it. Highly informal, but I don't know any technique for formalizing it better.

Comment author: Kaj_Sotala 14 March 2009 11:55:24PM 20 points [-]

Without taking a stance on the question of child sexuality - what you say is true, but is there any particular selection pressure for it to be off, either? Evolution goes for the simplest solution, and "always on" seems to me simpler than "off until a specific age, then on".

Of course, that's an oversimplification. The required machinery may simply not be developed yet, in the same way that you need to first grow to be four feet tall before you can grow to be five feet tall. But then, when you reach the size of four feet, you already have four fifths of your five feet-tallness in place, so it stands to reason that that at least part of what makes sex pleasurable will be in place before adolescence. Whether it's active is obviously a separate question, but I don't think "has no particular evolutionary reason to be active" tells us much by itself.

Comment author: Yvain 15 March 2009 12:38:51AM *  4 points [-]

I could be confusing Freudian stuff with real experimental results, but I seem to remember that children go through a stage up until about 6 where they're somewhat sexual, and then between that age and puberty the sex drive switches off or even into full reverse. This is the reason that young boys tend to think girls have cooties and are gross, and vice versa. It's evolution's way of saying "Not yet".

In response to Closet survey #1
Comment author: scientism 14 March 2009 07:21:46PM 13 points [-]

I don't think people have (ethical) value simply because they exist. I think they should have to do a lot more than that before I should have to care whether they live or die.

In response to comment by scientism on Closet survey #1
Comment author: Yvain 15 March 2009 12:35:27AM 5 points [-]

So if I were to start torturing a random child, would you object? Assume the child has never done anything important to make him especially valuable.

In response to Closet survey #1
Comment author: Yvain 15 March 2009 12:29:34AM *  33 points [-]
  • I don't like libertarianism. It makes some really good points, and clearly there are lots of things government should stay out of, but the whole narrative of government as the evil villain that can never do anything right strikes me as more of a heroic myth than a useful way to shape policy. This only applies to libertarians who go overboard, though. I like Will Wilkinson, but I hate Lew Rockwell.

  • I think the better class of mystics probably know some things about the mind the rest of us don't. I tend to trust yogis who say they've achieved perfect bliss after years of meditation, although I think there's a neurological explanation (and would like to know what it is). I think Crowley's project to systematize and scientifically explain mysticism had some good results even though he did go utterly off the deep end.

  • I am not sure I will sign up for cryonics, although I am still seriously considering it. The probability of ending up immortal and stuck in a dystopia where I couldn't commit suicide scares me too much.

  • I have a very hard time going under 2-3% belief in anything that lots of other people believe. This includes religion, UFOs, and ESP. Not astrology though, oddly enough; I'll happily go so low on that one it'd take exponential notation to describe properly.

  • I like religion. I don't believe it, I just like it. Greek mythology is my favorite, but I think the Abrahamic religions are pretty neat too.

  • I am a very hard-core utilitarian, and happily accept John Maxwell's altruist argument. I sorta accept Torture vs. Dust Specks on a rational but not an emotional level.

  • I am still not entirely convinced that irrationality can't be fun. I sympathize with some of those Wiccans who worship their gods not because they believe in them but just because they like them. Of course, I separate this from belief in belief, which really is an evil.

In response to Closet survey #1
Comment author: mathemajician 15 March 2009 12:00:10AM 10 points [-]

I am an atheist who does not believe in the super natural. Great. Tons of evidence and well thought out reasoning on my side.

But... well... a few things have happened in my life that I find rather difficult to explain. I feel like a statistician looking at a data set with a nice normal distribution... and a few very low probability outliers. Did I just get a weird sample, or is something going on here? I figure that they are most likely to be just weird data points, but they are weird enough to bother me.

Let me give you one example. A few years ago I had a dream that I was eating and out of the blue I discovered a shard of glass in my mouth. The dream bothered me so much that I had a flash back to the dream the next day as I was walking down the road. For me that's extremely unusual. It's rare that I can even remember a dream, and when I do they certainly don't bother me the next day. So, the day after that I was eating a salad and crunch. I spat out what was in my mouth and there was a seriously nasty looking slither of glass. I didn't cut my mouth or anything, no harm done. I just hit it with my tooth.

To the best of my knowledge that was the only time I've ever found glass in something I was eating, and it was the only time I've had a vivid dream about it that bothered me the next day (or any dream about it all). I didn't have any particular glass eating phobia before all this took place (except for a normal aversion to the idea), and I haven't been worried about it since (ok, except for looking rather carefully at salads from that particular cafeteria for a few weeks afterwards). Was this all just a really weird coincidence? As far as I can make out the probabilities are just too low to be ignored. To make matters worse, I have a few other stories that I find just as difficult to explain away as coincidence.

Now, I wouldn't say that I "believe" that something seriously weird is going on here. That would be much too strong. However, because I don't feel that I can adequately account for some of my observations of the world, I think I must assign a small probability that there is something very seriously strange going on in the universe and that these events were not weird flukes.

I have other things to say but that would get into topics currently banned from this blog :-/

Comment author: Yvain 15 March 2009 12:14:07AM 6 points [-]

The Skeptic's Trilemma

58 Post author: Yvain 15 March 2009 12:12AM

Followup to: Talking Snakes: A Cautionary Tale

Related to: Explain, Worship, Ignore

Skepticism is like sex and pizza: when it's good, it's very very good, and when it's bad, it's still pretty good.

It really is hard to dislike skeptics. Whether or not their rational justifications are perfect, they are doing society a service by raising the social cost of holding false beliefs. But there is a failure mode for skepticism. It's the same as the failure mode for so many other things: it becomes a blue vs. green style tribe, demands support of all 'friendly' arguments, enters an affective death spiral, and collapses into a cult.

What does it look like when skepticism becomes a cult? Skeptics become more interested in supporting their "team" and insulting the "enemy" than in finding the truth or convincing others. They begin to think "If a assigning .001% probability to Atlantis and not accepting its existence without extraordinarily compelling evidence is good, then assigning 0% probability to Atlantis and refusing to even consider any evidence for its existence must be great!" They begin to deny any evidence that seems pro-Atlantis, and cast aspersions on the character of anyone who produces it. They become anti-Atlantis fanatics.

Wait a second. There is no lost continent of Atlantis. How do I know what a skeptic would do when confronted with evidence for it? For that matter, why do I care?

Way back in 2007, Eliezer described the rationalist equivalent of Abort, Retry, Fail: the trilemma of Explain, Worship, Ignore. Don't understand where rain comes from? You can try to explain it as part of the water cycle, although it might take a while. You can worship it as the sacred mystery of the rain god. Or you can ignore it and go on with your everyday life.

So someone tells you that Plato, normally a pretty smart guy, wrote a long account of a lost continent called Atlantis complete with a bunch of really specific geographic details that seem a bit excessive for a meaningless allegory. Plato claims to have gotten most of the details from a guy called Solon, legendary for his honesty, who got them from the Egyptians, who are known for their obsessive record-keeping. This seems interesting. But there's no evidence for a lost continent anywhere near the Atlantic Ocean, and geology tells us continents can't just go missing.

One option is to hit Worship. Between the Theosophists, Edgar Cayce, the Nazis, and a bunch of well-intentioned but crazy amateurs including a U.S. Congressman, we get a supercontinent with technology far beyond our wildest dreams, littered with glowing crystal pyramids and powered by the peaceful and eco-friendly mystical wisdom of the ancients, source of all modern civilization and destined to rise again to herald the dawning of the Age of Aquarius.

Or you could hit Ignore. I accuse the less pleasnt variety of skeptic of taking this option. Atlantis is stupid. Anyone who believes it is stupid. Plato was a dirty rotten liar. Any scientist who finds anomalous historical evidence suggesting a missing piece to the early history of the Mediterranean region is also a dirty rotten liar, motivated by crazy New Age beliefs, and should be fired. Anyone who talks about Atlantis is the Enemy, and anyone who denies Atlantis gains immediate access to our in-group and official Good Rational Scientific Person status.

Spyridon Marinatos, a Greek archaeologist who really deserves more fame than he received, was a man who hit Explain. The geography of Plato's Atlantis, a series of concentric circles of land and sea, had been derided as fanciful; Marinatos noted1 that it matched the geography of the Mediterranean island of Santorini quite closely. He also noted that Santorini had a big volcano right in the middle and seemed somehow linked to the Minoan civilization, a glorious race of seafarers who had mysteriously collapsed a thousand years before Plato. So he decided to go digging in Santorini. And he found...

...the lost city of Atlantis. Well, I'm making an assumption here. But the city he found was over four thousand years old, had a population of over ten thousand people at its peak, boasted three-story buildings and astounding works of art, and had hot and cold running water - an unheard-of convenience that it shared with the city in Plato's story. For the Early Bronze Age, that's darned impressive. And like Plato's Atlantis, it was destroyed in a single day. The volcano that loomed only a few miles from its center went off around 1600 BC, utterly burying it and destroying its associated civilization. No one knows what happened to the survivors, but the most popular theory is that some fled to Egypt2, with which the city had flourishing trade routes at its peak.

The Atlantis = Santorini equivalence is still controversial, and the point of this post isn't to advocate for it. But just look at the difference between Joe Q. Skeptic and Dr. Marinatos. Both were rightly skeptical of the crystal pyramid story erected by the Atlantis-worshippers. But Joe Q. Skeptic considered the whole issue a nuisance, or at best a way of proving his intellectual superiority over the believers. Dr. Marinatos saw an honest mystery, developed a theory that made testable predictions, then went out and started digging.

The fanatical skeptic, when confronted with some evidence for a seemingly paranormal claim, says "Wow, that's stupid." It's a soldier on the opposing side, and the only thing to be done with it is kill it as quickly as possible. The wise skeptic, when confronted with the same evidence, says "Hmmm, that's interesting."

Did people at Roswell discovered the debris of a strange craft made of seemingly otherworldly material lying in a field, only to be silenced by the government later? You can worship the mighty aliens who are cosmic bringers of peace. You can ignore it, because UFOs don't exist so the people are clearly lying. Or you can search for an explanation until you find that the government was conducting tests of Project Mogul in that very spot.

Do thousands of people claim that therapies with no scientific basis are working? You can worship alternative medicine as a natural and holistic alternative to stupid evil materialism. You can ignore all the evidence for their effectiveness. Or you can shut up and discover the placebo effect, explaining the lot of them in one fell swoop.

Does someone claim to see tiny people, perhaps elves, running around and doing elvish things? You can call them lares and worship them as household deities. You can ignore the person because he's an obvious crank. Or you can go to a neurologist, and he'll explain that the person's probably suffering from Charles Bonnet Syndrome.

All unexplained phenomena are real. That is, they're real unexplained phenomena. The explanation may be prosaic, like that people are gullible. Or it may be an entire four thousand year old lost city of astounding sophistication. But even "people are gullible" can be an interesting explanation if you're smart enough to make it one. There's a big difference between "people are gullible, so they believe in stupid things like religion, let's move on" and a complete list of the cognitive biases that make explanations involving agency and intention more attractive than naturalistic explanations to a naive human mind. A sufficiently intelligent thinker could probably reason from the mere existence of religion all the way back to the fundamentals of evolutionary psychology.

This I consider a specific application of a more general rationalist technique: not prematurely dismissing things that go against your worldview. There's a big difference between dismissing that whole Lost Continent of Atlantis story, and prematurely dismissing it. It's the difference between discovering an ancient city and resting smugly satisfied that you don't have to.

 

Footnotes

1: I may be unintentionally sexing up the story here. I read a book on Dr. Marinatos a few years ago, and I know he did make the Santorini-Atlantis connection, but I don't remember whether he made it before starting his excavation, or whether it only clicked during the dig (and the Internet is silent on the matter). If it was the latter, all of my moralizing about how wonderful it was that he made a testable prediction falls a bit flat. I should have used another example where I knew for sure, but this story was too perfect. Mea culpa.

2: I don't include it in the main article because it is highly controversial and you have to fudge some dates for it to really work out, but here is a Special Bonus Scientific Explanation of a Paranormal Claim: the eruption of this same supervolcano in 1600 BC caused the series of geologic and climatological catastrophes recorded in the Bible as the Ten Plagues of Egypt. However, I specify that I'm including this because it's fun to think about rather than because there's an especially large amount of evidence for it.

The Least Convenient Possible World

165 Post author: Yvain 14 March 2009 02:11AM

Related to: Is That Your True Rejection?

"If you’re interested in being on the right side of disputes, you will refute your opponents’ arguments.  But if you’re interested in producing truth, you will fix your opponents’ arguments for them.  To win, you must fight not only the creature you encounter; you must fight the most horrible thing that can be constructed from its corpse."

   -- Black Belt Bayesian, via Rationality Quotes 13

Yesterday John Maxwell's post wondered how much the average person would do to save ten people from a ruthless tyrant. I remember asking some of my friends a vaguely related question as part of an investigation of the Trolley Problems:

You are a doctor in a small rural hospital. You have ten patients, each of whom is dying for the lack of a separate organ; that is, one person needs a heart transplant, another needs a lung transplant, another needs a kidney transplant, and so on. A traveller walks into the hospital, mentioning how he has no family and no one knows that he's there. All of his organs seem healthy. You realize that by killing this traveller and distributing his organs among your patients, you could save ten lives. Would this be moral or not?

I don't want to discuss the answer to this problem today. I want to discuss the answer one of my friends gave, because I think it illuminates a very interesting kind of defense mechanism that rationalists need to be watching for. My friend said:

It wouldn't be moral. After all, people often reject organs from random donors. The traveller would probably be a genetic mismatch for your patients, and the transplantees would have to spend the rest of their lives on immunosuppressants, only to die within a few years when the drugs failed.

On the one hand, I have to give my friend credit: his answer is biologically accurate, and beyond a doubt the technically correct answer to the question I asked. On the other hand, I don't have to give him very much credit: he completely missed the point and lost a valuable effort to examine the nature of morality.

So I asked him, "In the least convenient possible world, the one where everyone was genetically compatible with everyone else and this objection was invalid, what would you do?"

He mumbled something about counterfactuals and refused to answer. But I learned something very important from him, and that is to always ask this question of myself. Sometimes the least convenient possible world is the only place where I can figure out my true motivations, or which step to take next. I offer three examples:

 

1:  Pascal's Wager. Upon being presented with Pascal's Wager, one of the first things most atheists think of is this:

Perhaps God values intellectual integrity so highly that He is prepared to reward honest atheists, but will punish anyone who practices a religion he does not truly believe simply for personal gain. Or perhaps, as the Discordians claim, "Hell is reserved for people who believe in it, and the hottest levels of Hell are reserved for people who believe in it on the principle that they'll go there if they don't."

This is a good argument against Pascal's Wager, but it isn't the least convenient possible world. The least convenient possible world is the one where Omega, the completely trustworthy superintelligence who is always right, informs you that God definitely doesn't value intellectual integrity that much. In fact (Omega tells you) either God does not exist or the Catholics are right about absolutely everything.

Would you become a Catholic in this world? Or are you willing to admit that maybe your rejection of Pascal's Wager has less to do with a hypothesized pro-atheism God, and more to do with a belief that it's wrong to abandon your intellectual integrity on the off chance that a crazy deity is playing a perverted game of blind poker with your eternal soul?

2: The God-Shaped Hole. Christians claim there is one in every atheist, keeping him from spiritual fulfillment.

Some commenters on Raising the Sanity Waterline don't deny the existence of such a hole, if it is intepreted as a desire for purpose or connection to something greater than one's self. But, some commenters say, science and rationality can fill this hole even better than God can.

What luck! Evolution has by a wild coincidence created us with a big rationality-shaped hole in our brains! Good thing we happen to be rationalists, so we can fill this hole in the best possible way! I don't know - despite my sarcasm this may even be true. But in the least convenient possible world, Omega comes along and tells you that sorry, the hole is exactly God-shaped, and anyone without a religion will lead a less-than-optimally-happy life. Do you head down to the nearest church for a baptism? Or do you admit that even if believing something makes you happier, you still don't want to believe it unless it's true?

3: Extreme Altruism. John Maxwell mentions the utilitarian argument for donating almost everything to charity.

Some commenters object that many forms of charity, especially the classic "give to starving African orphans," are counterproductive, either because they enable dictators or thwart the free market. This is quite true.

But in the least convenient possible world, here comes Omega again and tells you that Charity X has been proven to do exactly what it claims: help the poor without any counterproductive effects. So is your real objection the corruption, or do you just not believe that you're morally obligated to give everything you own to starving Africans?

 

You may argue that this citing of convenient facts is at worst a venial sin. If you still get to the correct answer, and you do it by a correct method, what does it matter if this method isn't really the one that's convinced you personally?

One easy answer is that it saves you from embarrassment later. If some scientist does a study and finds that people really do have a god-shaped hole that can't be filled by anything else, no one can come up to you and say "Hey, didn't you say the reason you didn't convert to religion was because rationality filled the god-shaped hole better than God did? Well, I have some bad news for you..."

Another easy answer is that your real answer teaches you something about yourself. My friend may have successfully avoiding making a distasteful moral judgment, but he didn't learn anything about morality. My refusal to take the easy way out on the transplant question helped me develop the form of precedent-utilitarianism I use today.

But more than either of these, it matters because it seriously influences where you go next.

Say "I accept the argument that I need to donate almost all my money to poor African countries, but my only objection is that corrupt warlords might get it instead", and the obvious next step is to see if there's a poor African country without corrupt warlords (see: Ghana, Botswana, etc.) and donate almost all your money to them. Another acceptable answer would be to donate to another warlord-free charitable cause like the Singularity Institute.

If you just say "Nope, corrupt dictators might get it," you may go off and spend the money on a new TV. Which is fine, if a new TV is what you really want. But if you're the sort of person who would have been convinced by John Maxwell's argument, but you dismissed it by saying "Nope, corrupt dictators," then you've lost an opportunity to change your mind.

So I recommend: limit yourself to responses of the form "I completely reject the entire basis of your argument" or "I accept the basis of your argument, but it doesn't apply to the real world because of contingent fact X." If you just say "Yeah, well, contigent fact X!" and walk away, you've left yourself too much wiggle room.

In other words: always have a plan for what you would do in the least convenient possible world.

In response to Soulless morality
Comment author: billswift 14 March 2009 11:10:05PM *  1 point [-]

Life, as such, is valueless.

Every person's life has value to that person (we can reasonably assume - or he would commit suicide, either actively or passively).

I have long believed that the rational value of a person's life to other people is the contribution that person makes to the welfare of the others - either directly to friends and family or to society (through paid work) to strangers. I stress paid work, since there is no way to judge whether someone's unpaid actions is a net contribution or not - at least if someone is willing to pay for something it is a benefit to that person.

Part of the problem you discuss is a confusion between a person's self-valuation and his value to others.

Comment author: Yvain 15 March 2009 12:07:50AM 8 points [-]

I don't really buy that. As a wise man once said "We were put here to serve others. Why others were put here is beyond me."

If the only value is to help others, and the others' only value is to help you, then isn't the whole system ultimately valueless?

In response to Soulless morality
Comment author: Yvain 15 March 2009 12:04:27AM *  15 points [-]

First, I agree with the main thrust of your argument: that our current morality is riddled to the core with things derived from supernatural beliefs, and we're going to have to rebuild it from the ground up. And that we need to stop using "life" as a primitive unit.

But I disagree with you about your specific example. I don't think those of us who want to donate money to poor countries are trying to create more human lives, or even save more human lives - see Parfit's Repugnant Conclusion. We're trying to alleviate suffering.

If the best way to alleviate suffering is to buy condoms for starving Africans to prevent the birth of another generation of starving Africans, I am all for this even though it decreases the number of Africans. If the best way to alleviate suffering is to try to improve the African economy through programs like microfinance, I am all for this even though it holds the number of Africans constant. If the best way to alleviate suffering is by curing malaria, I am all for this even though it increases the number of Africans - as long as the total amount of suffering including those new Africans is less than it was before.

I draw a sharp distinction between about twenty different meanings of the word "value". Value on instrumental grounds is one of them. When I say that drug addicts are less valuable than other people, I probably mean they're less useful to society. That doesn't mean it's morally more okay to torture a drug addict than it is to torture Bill Gates (except insofar as torturing Bill Gates would disrupt his various societally useful activities). But there are a whole lot of values that aren't value to society - even someone who's valueless to society may well have a lot of value to himself.

I don't know if this is exactly what you're proposing, but I'll argue against it anyway - I reject a "multiplier" theory of value. That is, if Person A is twice as good a person as Person B (more intelligent, prettier, whatever) then that doesn't mean that torturing Person B is twice as acceptable as torturing Person A. A unit of suffering is a unit of suffering. Person A deserves credit for all the great things he does, but that doesn't change the ethical calculus. The exception is that it's better to kill an unhappy person than a happy person, because it's a better change to the joy/suffering balance. You still can't go around killing unhappy people willy-nilly though because of precedent reasons.

Although I value rationality, I have to admit that I value it mostly as an instrumental value. Although I value it as a terminal value a little, I don't think it has quite as much power for me as the joy vs. suffering value. I can't think of any non-trivial amount of torture I would inflict on Person A that would be justified if it caused Person B to read a book on Bayesian statistics. That's why I'd prefer to spend my money on starving people in Africa than anything else. It seems like the cheapest way to alleviate the most suffering, and alleviating suffering is top priority in my ethical system right now (I include in this really indirect ways to help people in Africa, like the Singularity Institute).

My comment isn't as clear as I think it should be, but I'm not even sure to what degree we disagree so I won't fret too much about it (I might just be rounding you off to the nearest cliche, as one person put it). One thing, though: you do accept that even if you value education more than saving starving Africans, John Maxwell's argument still holds, right? You just need to donate all that money to educational charities. The argument holds as long as there's something, anything, you value more than your own convenience.

Question: am I the only person who (unless I considered immortality likely) would vastly, VASTLY prefer a death sentence to twenty years in prison?

Comment author: nazgulnarsil 14 March 2009 10:05:50AM 2 points [-]

with regards to the third question: what if I believe that any resources given simply allow the population to expand and hence cause more suffering than letting people die?

Comment author: Yvain 14 March 2009 11:14:31AM *  13 points [-]

If you don't really believe that, and it's just your excuse for not giving away lots of money, you should say loud and clear "I don't believe I'm morally obligated to reduce suffering if it inconveniences me too much." And then you've learned something useful about yourself.

But if you do really believe that, and you otherwise accept John's argument, you should say explicitly, "I accept I'm morally obligated to reduce suffering as much as possible, even at the cost of great inconvenience to myself. However, I am worried because of the contingent fact that giving people more resources will lead to more population, causing more suffering."

And if you really do believe that and think it through, you'll end up spending almost all your income on condoms for third world countries.

Comment author: MBlume 14 March 2009 02:21:30AM *  38 points [-]

I'm not sure if I'm evading the spirit of the post, but it seems to me that the answer to the opening problem is this:

If you were willing to kill this man to save these ten others, then you should long ago have simply had all ten patients agree to a 1/10 game of Russian Roulette, with the proviso that the nine winners get the organs of the one loser.

Comment author: Yvain 14 March 2009 02:28:53AM 16 points [-]

While emphasizing that I don't want this post to turn into a discussion of trolley problems, I endorse that solution.

The Least Convenient Possible World

165 Post author: Yvain 14 March 2009 02:11AM

Related to: Is That Your True Rejection?

"If you’re interested in being on the right side of disputes, you will refute your opponents’ arguments.  But if you’re interested in producing truth, you will fix your opponents’ arguments for them.  To win, you must fight not only the creature you encounter; you must fight the most horrible thing that can be constructed from its corpse."

   -- Black Belt Bayesian, via Rationality Quotes 13

Yesterday John Maxwell's post wondered how much the average person would do to save ten people from a ruthless tyrant. I remember asking some of my friends a vaguely related question as part of an investigation of the Trolley Problems:

You are a doctor in a small rural hospital. You have ten patients, each of whom is dying for the lack of a separate organ; that is, one person needs a heart transplant, another needs a lung transplant, another needs a kidney transplant, and so on. A traveller walks into the hospital, mentioning how he has no family and no one knows that he's there. All of his organs seem healthy. You realize that by killing this traveller and distributing his organs among your patients, you could save ten lives. Would this be moral or not?

I don't want to discuss the answer to this problem today. I want to discuss the answer one of my friends gave, because I think it illuminates a very interesting kind of defense mechanism that rationalists need to be watching for. My friend said:

It wouldn't be moral. After all, people often reject organs from random donors. The traveller would probably be a genetic mismatch for your patients, and the transplantees would have to spend the rest of their lives on immunosuppressants, only to die within a few years when the drugs failed.

On the one hand, I have to give my friend credit: his answer is biologically accurate, and beyond a doubt the technically correct answer to the question I asked. On the other hand, I don't have to give him very much credit: he completely missed the point and lost a valuable effort to examine the nature of morality.

So I asked him, "In the least convenient possible world, the one where everyone was genetically compatible with everyone else and this objection was invalid, what would you do?"

He mumbled something about counterfactuals and refused to answer. But I learned something very important from him, and that is to always ask this question of myself. Sometimes the least convenient possible world is the only place where I can figure out my true motivations, or which step to take next. I offer three examples:

 

1:  Pascal's Wager. Upon being presented with Pascal's Wager, one of the first things most atheists think of is this:

Perhaps God values intellectual integrity so highly that He is prepared to reward honest atheists, but will punish anyone who practices a religion he does not truly believe simply for personal gain. Or perhaps, as the Discordians claim, "Hell is reserved for people who believe in it, and the hottest levels of Hell are reserved for people who believe in it on the principle that they'll go there if they don't."

This is a good argument against Pascal's Wager, but it isn't the least convenient possible world. The least convenient possible world is the one where Omega, the completely trustworthy superintelligence who is always right, informs you that God definitely doesn't value intellectual integrity that much. In fact (Omega tells you) either God does not exist or the Catholics are right about absolutely everything.

Would you become a Catholic in this world? Or are you willing to admit that maybe your rejection of Pascal's Wager has less to do with a hypothesized pro-atheism God, and more to do with a belief that it's wrong to abandon your intellectual integrity on the off chance that a crazy deity is playing a perverted game of blind poker with your eternal soul?

2: The God-Shaped Hole. Christians claim there is one in every atheist, keeping him from spiritual fulfillment.

Some commenters on Raising the Sanity Waterline don't deny the existence of such a hole, if it is intepreted as a desire for purpose or connection to something greater than one's self. But, some commenters say, science and rationality can fill this hole even better than God can.

What luck! Evolution has by a wild coincidence created us with a big rationality-shaped hole in our brains! Good thing we happen to be rationalists, so we can fill this hole in the best possible way! I don't know - despite my sarcasm this may even be true. But in the least convenient possible world, Omega comes along and tells you that sorry, the hole is exactly God-shaped, and anyone without a religion will lead a less-than-optimally-happy life. Do you head down to the nearest church for a baptism? Or do you admit that even if believing something makes you happier, you still don't want to believe it unless it's true?

3: Extreme Altruism. John Maxwell mentions the utilitarian argument for donating almost everything to charity.

Some commenters object that many forms of charity, especially the classic "give to starving African orphans," are counterproductive, either because they enable dictators or thwart the free market. This is quite true.

But in the least convenient possible world, here comes Omega again and tells you that Charity X has been proven to do exactly what it claims: help the poor without any counterproductive effects. So is your real objection the corruption, or do you just not believe that you're morally obligated to give everything you own to starving Africans?

 

You may argue that this citing of convenient facts is at worst a venial sin. If you still get to the correct answer, and you do it by a correct method, what does it matter if this method isn't really the one that's convinced you personally?

One easy answer is that it saves you from embarrassment later. If some scientist does a study and finds that people really do have a god-shaped hole that can't be filled by anything else, no one can come up to you and say "Hey, didn't you say the reason you didn't convert to religion was because rationality filled the god-shaped hole better than God did? Well, I have some bad news for you..."

Another easy answer is that your real answer teaches you something about yourself. My friend may have successfully avoiding making a distasteful moral judgment, but he didn't learn anything about morality. My refusal to take the easy way out on the transplant question helped me develop the form of precedent-utilitarianism I use today.

But more than either of these, it matters because it seriously influences where you go next.

Say "I accept the argument that I need to donate almost all my money to poor African countries, but my only objection is that corrupt warlords might get it instead", and the obvious next step is to see if there's a poor African country without corrupt warlords (see: Ghana, Botswana, etc.) and donate almost all your money to them. Another acceptable answer would be to donate to another warlord-free charitable cause like the Singularity Institute.

If you just say "Nope, corrupt dictators might get it," you may go off and spend the money on a new TV. Which is fine, if a new TV is what you really want. But if you're the sort of person who would have been convinced by John Maxwell's argument, but you dismissed it by saying "Nope, corrupt dictators," then you've lost an opportunity to change your mind.

So I recommend: limit yourself to responses of the form "I completely reject the entire basis of your argument" or "I accept the basis of your argument, but it doesn't apply to the real world because of contingent fact X." If you just say "Yeah, well, contigent fact X!" and walk away, you've left yourself too much wiggle room.

In other words: always have a plan for what you would do in the least convenient possible world.

Comment author: Yvain 14 March 2009 12:34:00AM *  40 points [-]

A lot of dojos preserve to some degree the social standards of Eastern countries where the sensei's sensei came from. And in Eastern countries, it's much less acceptable to try to question your teacher, or change things, or rock the boat, or show any form of weakness. I taught school in Japan for a while, and the first thing I learned was that naively asking "Any questions?" or "Any opinions on this?" or even "Anyone not understand?" was a waste of time.

Western cultures are a lot better at this, but not ideal. There's still pressure not to be the one person who asks all the questions all the time, and there's pressure not to say anything controversial out of the blue because you lose more status if you're wrong than you gain if you're right. I think part of the problem is that there really are dumb or egotistical people who, if given the chance will protest that they know a much better way to do everything and will waste the time of everyone else, and our society's decided to .make a devil's bargain to keep them under control.

The best solution to this is to found a new culture, live isolated from the rest of the world for a century developing different cultural norms, and then start the rationality dojo there. Of possible second-best solutions:

  • My Favorite Liar. Tell people that you're going to make X deliberately incorrect statements every training session and they've got to catch them.

  • Clickers. One of my lecturers uses these devices sort of like remote controls. You can input information into them and it gets sent wirelessly and anonymously to the lecturer's laptop. The theory is that if he says "Raise your hand if you don't understand this" or even "...if you disagree with this", no one will, but if he says "Enter whether or not you understand this into your clicker" he may get three or four "don't understand" responses. Anonymous suggestion boxes are a low-tech form of the same principle.

  • I always found the concept of Crocker's Rules very interesting. I also remember hearing of a community (wish I could remember which) in which it was absolutely forbidden to give negative feedback under certain circumstances, and the odd social dynamics that created. In a dojo-like setting, there might be situations when either of these two rules could be ritually enacted - for example, a special Crocker Hat, such that anyone wearing that hat was known to be under Crocker's Rules, and a special No Negative Feedback Hat (but with a flashier name, like White Crane Hat of Social Invincibility), which someone could wear when questioning the master or something and be absolutely immune to any criticism.

Comment author: AnnaSalamon 13 March 2009 07:06:19PM *  24 points [-]

This "art of kicking" is what pjeby has been working toward, AFAICT. I haven't read much of his writing, though. But an "art of kicking" would be a great thing to mix in with the OB/LW corpus, if pjeby has something that works, which I think he has at least some of -- and if we and he can figure out how to hybridize kicking research and training with punching research and training.

I'd also love to bring in more people from the entrepreneurship/sales/marketing communities. I've been looking at some of their better literature, and it has rationality techniques (techniques for not shooting yourself in the foot by wishful thinking, overconfidence, etc.) and get-things-done techniques mixed together. I love the sit-and-think math nerd types too, and we need sitting and thinking; the world is full of people taking action toward the wrong goals. But I'd expect better results from our rationalist community if we mixed in more people whose natural impulses were toward active experiments and short-term visible results.

Comment author: Yvain 13 March 2009 11:04:30PM *  19 points [-]

Pjeby's working on akrasia? I'll have to check out his site.

That brings up a related question that I think Eliezer hinted at: what pre-existing bodies of knowledge can we search through for powerful techniques so that we don't have to re-invent the wheel? Entrepreneurship stuff is one. Lots of people have brought up pick-up artists and poker, so those might be others.

I nominate a fourth that may be controversial: mysticism. Not the "summon demons" style of mysticism, but yoga and Zen and related practices. These people have been learning how to examine/quiet/rearrange their minds and sort out the useful processes from the useless processes for the past three thousand years. Even if they've been working off crazy metaphysics, it'd be surprising if they didn't come up with something. Eliezer talks in mystical language sometimes, but I don't know whether that's because he's studied and approves of mysticism or just likes the feel of it.

What all of these things need is a testing process combined with people who are already high-level enough that they can sort through all the dross and determine which techniques are useful without going native or opening themselves up to the accusation that they're doing so; ie people who can sort through the mystical/pick-up artist/whatever literature and separate out the things that are useful to rationalists from the things specific to a certain worldview hostile to our own. I've seen a few good people try this, but it's a mental minefield and they tend to end up "going native".

Comment author: Vladimir_Nesov 13 March 2009 01:41:23AM *  1 point [-]

You may well have an irrational tendency to not expect things that turn out to be real. There is an unfathomable number of things that you don't expect to find, so the choosing of your next dragon needs to be performed based on some kind of evidence, like a known bias.

P.S. Nick's interpretation is correct: in "expect" I included very unlikely but still barely plausible claims.

Comment author: Yvain 13 March 2009 02:57:12AM 3 points [-]

Why assume Roland assigns epsilon probability to miracles? If he's sufficiently uncertain about atheism so as to want to look for miracles, I'd say going out and looking is the best possible thing he can be doing. It's too much to expect that he go from being a Christian to having epsilon probability of God right away. If a few years after being a church-going Christian he's now at 5% probability of God, I wouldn't say he's doing anything wrong. Even Jeffreyssai doesn't say every decision should be made in less than a second. Just don't take thirty years.

It could be argued that because of the importance of miracles (a single one would prove some form of religion, religion is a very important issue) it's worth keeping an eye out for them even at such low levels of expectation that you'd give up looking for, say, the Higgs boson.

If thirty years from now he's still looking for miracles, that would be a problem, but the thought-mode that one should look for evidence if one is uncertain wouldn't be the issue. The issue would be whatever it was that was keeping his probability distribution at 5%.

Comment author: cleonid 13 March 2009 02:09:47AM 4 points [-]

" I think theism is wrong. "

I believe you are likely to be right.

"I think it can be demonstrated to be wrong on logical grounds."

I'm really intrigued. How?

Comment author: Yvain 13 March 2009 02:30:05AM *  8 points [-]

Uh oh.

http://www.overcomingbias.com/2007/08/religions-claim.html and http://yudkowsky.net/rational/technical , but a full argument would probably take forever, be inappropriate for this site, and have me answering religious objections until the cows came home.

Logical was probably not as good a word as "rational" here. If saying "on rational grounds" makes more sense than "on logical grounds", feel free to replace it.

If you're really really really interested, send me your email and I'll try to sketch out some thoughts, but beyond what's in the two links above I doubt anything I say will be that much more exciting than anything you've heard atheists say before.

Comment author: anonym 13 March 2009 02:20:15AM *  15 points [-]

What the lady in Cairo regarded as absurd (a monkey having a human baby) has almost no relation to what educated people who believe in evolution actually believe. What Bill Mahers regarded as absurd (a talking snake) is exactly what many Christians actually believe. The two assertions of absurdity are therefore not alike in the way that you suggest they are.

I agree with your underlying point about the absurdity heuristic not working well, but do any of us not realize this already given what modern physics tells us of the universe we live in?

Comment author: Yvain 13 March 2009 02:25:44AM 31 points [-]

Many (most? all?) Christians believe the snake was really Satan, who took the form of a snake to trick Eve. Treating it as an ordinary snake that happened to be able to talk is probably as gross a misrepresentation as the lady's misrepresentation of evolution.

Comment author: Yvain 13 March 2009 02:20:39AM *  72 points [-]

Eliezer, I have recommended to you before that you read The Darkness That Comes Before and the associated trilogy. I repeat that recommendation now. The monastery of Ishual is your rationalist dojo, and Anasurimbor Kellhus is your beisutsukai surrounded by a visible aura of formidability. The book might even give you an idea or two.

My only worry with the idea of these dojos is that I doubt the difference between us and Anasurimbor Kellhus is primarily a difference in rationality levels. I think it is more likely to be akrasia. Even an irrational, downright stupid person can probably think of fifty ways to improve his life, most of which will work very well if he only does them (quit smoking, quit drinking, study harder in school, go on a diet). And a lot of people with pretty well developed senses of rationality whom I know, don't use them for anything more interesting than winning debates about abortion or something. Maybe the reason rationalists rarely do that much better than anyone else is that they're not actually using all that extra brainpower they develop. The solution to that isn't more brainpower.

Kellhus was able to sit down, enter the probability trance, decide on the best course of action for the immediate future, and just go do it. When I tried this, I never found the problem was in the deciding - it doesn't take a formal probability trance to chart a path through everyday life - it was in following the results. Among the few Kellhus-worthy stories I've ever heard from reality was you deciding the Singularity was the most important project, choosing to devote your life to it, and not having lost that resolve fifteen years later. If you could bottle that virtue, it would be worth more than the entire Bayesian corpus combined. I don't doubt that it's positively correlated with rationality, but I do doubt it's a 1 or even .5 correlation.

Comment author: Eliezer_Yudkowsky 13 March 2009 01:23:16AM 6 points [-]

Simply invest the $3000 at 5% interest, and give the $150/year to the Africans

to the Africans' warlords, who steal the money and use it to stay in power

Fixed that for you.

As P. J. O'Rourke says, speaking of something like a total of $200,000(?) per poor person spent by the American welfare system, it's a flabbergasting phenomenon that appears to be real: You can't fix poverty by giving people money.

I'd love it if the Singularity Institute had an endowment by the time you get out of school, but I wouldn't count on it if I were me.

Comment author: Yvain 13 March 2009 01:48:42AM *  9 points [-]

"to the Africans' warlords, who steal the money and use it to stay in power"

Note the reference to Jeffrey Sachs in my comment. If you haven't read The End of Poverty, he demolishes the "It's all warlords stealing the money" argument pretty darned thoroughly.

I was thinking of my prime giving years as late middle age, two or three decades down the line, and I was hoping less that you would have an endowment than that, you know, you would control the world and dazzle the few remaining people who hadn't advanced to a Stross-ian Economy 2.0 by transforming small asteroids into giant gold nuggets. But I guess an endowment would be nice too.

Talking Snakes: A Cautionary Tale

107 Post author: Yvain 13 March 2009 01:41AM

I particularly remember one scene from Bill Maher's "Religulous". I can't find the exact quote, but I will try to sum up his argument as best I remember.

Christians believe that sin is caused by a talking snake. They may have billions of believers, thousands of years of tradition behind them, and a vast literature of apologetics justifying their faith - but when all is said and done, they're adults who believe in a talking snake.

I have read of the absurdity heuristic. I know that it is not carte blanche to go around rejecting beliefs that seem silly. But I was still sympathetic to the talking snake argument. After all...a talking snake?

I changed my mind in a Cairo cafe, talking to a young Muslim woman. I let it slip during the conversation that I was an atheist, and she seemed genuinely curious why. You've all probably been in such a situation, and you probably know how hard it is to choose just one reason, but I'd been reading about Biblical contradictions at the time and I mentioned the myriad errors and atrocities and contradictions in all the Holy Books.

Her response? "Oh, thank goodness it's that. I was afraid you were one of those crazies who believed that monkeys transformed into humans."

I admitted that um, well, maybe I sorta kinda might in fact believe that.

It is hard for me to describe exactly the look of shock on her face, but I have no doubt that her horror was genuine. I may have been the first flesh-and-blood evolutionist she ever met. "But..." she looked at me as if I was an idiot. "Monkeys don't change into humans. What on Earth makes you think monkeys can change into humans?"

I admitted that the whole process was rather complicated. I suggested that it wasn't exactly a Optimus Prime-style transformation so much as a gradual change over eons and eons. I recommended a few books on evolution that might explain it better than I could.

She said that she respected me as a person but that quite frankly I could save my breath because there was no way any book could possibly convince her that monkeys have human babies or whatever sort of balderdash I was preaching. She accused me and other evolution believers of being too willing to accept absurdities, motivated by our atheism and our fear of the self-esteem hit we'd take by accepting Allah was greater than ourselves.

It is not clear to me that this woman did anything differently than Bill Maher. Both heard statements that sounded so crazy as to not even merit further argument. Both recognized that there was a large group of people who found these statements plausible and had written extensive literature justifying them. Both decided that the statements were so absurd as to not merit examining that literature more closely. Both came up with reasons why they could discount the large number of believers because those believers must be biased.

I post this as a cautionary tale as we discuss the logic or illogic of theism. I propose taking from it the following lessons:

- The absurdity heuristic doesn't work very well.

- Even on things that sound really, really absurd.

- If a large number of intelligent people believe something, it deserves your attention. After you've studied it on its own terms, then you have a right to reject it. You could still be wrong, though.

- Even if you can think of a good reason why people might be biased towards the silly idea, thus explaining it away, your good reason may still be false.

- If someone cannot explain why something is not stupid to you over twenty minutes at a cafe, that doesn't mean it's stupid. It just means it's complicated, or they're not very good at explaining things.

- There is no royal road.

(special note to those prone to fundamental attribution errors: I do not accept theism. I think theism is wrong. I think it can be demonstrated to be wrong on logical grounds. I think the nonexistence of talking snakes is evidence against theism and can be worked into a general argument against theism. I just don't think it's as easy as saying "talking snakes are silly, therefore theism is false." And I find it embarrassing when atheists say things like that, and then get called on it by intelligent religious people.)

Comment author: Vladimir_Nesov 13 March 2009 12:34:35AM 1 point [-]

You should only look for miracles if you expect to find them. And naturalistic understanding of the world insists that you shouldn't expect that.

Comment author: Yvain 13 March 2009 12:48:33AM 5 points [-]

This view, taken to its logical extreme, would invalidate science. You're supposed to put some effort into looking for things that could disprove your current worldview.

Comment author: cleonid 13 March 2009 12:39:44AM 6 points [-]

“You're using rationalist methods to support religion”

Thank you very much for the compliment. However, it is totally undeserved. Being essentially an atheist (well, agnostic to be precise), supporting religion was the last thing on my mind. What I really wanted to do is to test how rational and intelligent people, which I hoped would be overrepresented on this forum, would react to arguments that go against their preferred view.

It is interesting that everyone seems to assume that I am a religious person myself, though I thought the contrary should be pretty obvious from the post title. Personally, I have yet to meet people who would call their beliefs “irrational”.

Comment author: Yvain 13 March 2009 12:45:09AM 2 points [-]
Comment author: Eliezer_Yudkowsky 13 March 2009 12:05:10AM *  16 points [-]

You need to choose a very carefully targeted charity that takes into account knock-on effects like displacing local production, before you should accept any such claim. It is now being suggested that the majority of aid is actually counterproductive and destroys local industry in exchange for giving Westerners a warm fuzzy feeling. Hence Africa staying poor despite (because of?) hundreds of billions spent.

Frankly, if you're going to put in that much work into buying expected utilons at the cheapest available price, you shouldn't be considering any mainstream charity targets like aid to developing countries. Let Bill Gates worry about it, if that's the limit of his creativity.

Comment author: Yvain 13 March 2009 12:23:32AM *  15 points [-]

I've done some research into this, and found a few targeted charities that I trust.

However, I'm a bit skeptical of the "aid is naturally counterproductive" claim (which I admit is stronger than what you're saying here, although I have heard some people say it). There are definitely some cases where it's true (you don't just send in random goods for free!), but the claim that you can't help poor people so you might as well keep all your money is just too convenient.

Take the claim from the OB post that it would take $3000 per person to raise per capita income $3. Unless they are referring to some specific, especially stupid kind of aid, this is clearly false. Simply invest the $3000 at 5% interest, and give the $150/year to the Africans and you've raised per capita income 50x as much. Not that this is a good idea, but it does seem to show there's something fishy about the calculation.

Jeffrey Sachs writes some interesting responses to the claim that Africa is too corrupt to be able to handle aid. If anyone has seen a specific counterargument to Sachs' claims, please link me to it.

Before I went back to school and lost my income, my favorite charities were microfinance, iodine supplementation (see Raising the World's IQ ) and yes, the Singularity Institute. Although I will be very disappointed if you guys (or anyone) still need money by the time I'm in my prime donating years.

Comment author: Hans 12 March 2009 01:30:05AM 70 points [-]

Actually, the trick worked, but the effects had worn off by the time you wrote this message, which is why you deny having your opinion on the AI issue completely reversed in a shocking aha-erlebnis, for a brief ten minutes at least. Remember to videotape yourself the next time.

Comment author: Yvain 13 March 2009 12:04:45AM 23 points [-]

The article made it clear that this would happen, but I never even considered it.

I conclude that possibly I was not as interested in trying the experiment as I thought, but rather wanted to be able to claim I was a good scientist who tests things that are easily testable. Good catch.

Comment author: Yvain 12 March 2009 11:59:56PM *  10 points [-]

There's an objection to be made that you can't be sure how useful charity is, but it's not a very strong objection. Last I heard there was pretty good evidence converging around $1000 or so being enough to save a person's life.

I accept this argument as valid and have for some time. I doubt I could donate everything to charity, but ~50% seems like a good compromise between human weakness and inhuman mathematics. We'll see how far I follow through on this when I get an income.

Comment author: Yvain 12 March 2009 11:47:24PM *  13 points [-]

I think this fails when you think of it on Bayesian grounds.

If we want to use the large number of intelligent religious people as evidence for the truth of religion, we need to show that P(intelligent believers|religion true) is greater than P(intelligent believers|religion false); that intelligent people are more likely to believe religion if it's true than if it's false.

But at most one religion can be true. Therefore, all other religions are false. But lots of people, many intelligent, believe both (for example) Christianity and Hinduism. If (for the sake of argument) we're wondering whether Christianity is true, then we cannot explain all the smart Hindus without admitting that people are just as likely to believe a false religion as a true one.

But if people are just as likely to believe a false religion as a true one, then lots of people believing a religion is no evidence that it is true.

There is an argument for shifting views towards general religious feelings, since that can't be disproven so easily. And I do shift my views a little in that direction. But remember that you can't double-shift. You have to shift them from the place they would be if no one believed in religion at all. IE, imagine a world where everyone was a scientific materialist, and imagine the credence you would give to this new hypothesis someone just dreamt up that the world was created by supernatural beings six thousand years ago. Then you can multiply that credence by whatever factor you want to use for the high level of belief people have in it. But the original credence is so vanishingly low that even the extra believers can't save it.

Also, re: Einstein and God. Richard Dawkins answers this better than I do. I suggest you read his work on the subject. The summary is that Einstein liked to use the word "God" as a metaphor for "physics", but wasn't a believer per se. Newton was, but he was also a believer in alchemy...

But I'd still like to thank you for bringing up this topic. You're using rationalist methods to support religion, which is exactly how all religious people should be doing it and which is something rationalists should take seriously. I'm a bit sad you're getting downvoted as much as you are, compared to some inane comments about how stupid all religious people must be that tend to be voted up. I predict most atheists will be overconfident in their rejection of religion, simply because most people are overconfident in any politically charged topic they feel strongly about. It's good to occasionally have to listen to intelligent rationalist arguments in favor or religion to avoid an echo chamber effect.

Comment author: Yvain 12 March 2009 01:38:21PM 3 points [-]

I'm completely uncertain whether this would work better, worse, or the same as more common methods of group decision-making. It's certainly an interesting idea.

I would make one caution, though. I find that businesses, schools, and decision-making workshops are far too willing to accept any cute or clever or radical sounding idea without any evidence that it works. It's easier to use them as an boast: "Don't say our decisions aren't rational. We care so much about being rational that we make all our decisions with special rationalist hats. If you're so rational, what do you do?" With "make decisions as well as possible based on available information" being a less acceptable answer than "have color coded teams using the Ten Step Rationalo-X Colors Method" or whatever.

For me to use this, I would need evidence that it worked. The best evidence would be assigning people to random groups, having one group talk it out informally and having the other use this hats method, and making them work on problems that are difficult but where there is one correct answer. If the hat people come to the correct answer more often than the non-hat people, then we use hats for everything.

I don't know why people don't do this more often for the common decision-making systems proposed, but I'll bet Robin Hanson would have some choice things to say about it.

Comment author: ciphergoth 11 March 2009 10:44:41PM 20 points [-]

Wouldn't squirting cold water in the left ear of creationists (or other healthy subjects who are having trouble letting go of a belief) be an effective test of Dr Ramachandran's hypothesis? And, potentially, a genuinely useful rationality technique?

I'm now imagining sneaking up on some stubbornly irrational people in my own life, water pistol in hand...

Comment author: Yvain 11 March 2009 11:07:25PM *  26 points [-]

It's a fair question, and if you read the article linked to under "posits two different reasoning modules", you know as much as I.

My thought is that there must be some reason this doesn't work, or else Ramachandran would have thought of it - he's famous for coming up with clever ways to test things other people thought were untestable (Google Ramachandran and synaesthesia for an example). Perhaps in normal life, the right hemisphere is as active as it's going to get, and if it hasn't overruled the left hemisphere already, it's not going to.

From the description of the technique, I think it's more complicated than just sticking water in the ear; I think it needs to go up into the vestibular system in the inner ear, which means it should probably only be done by a trained medical professional.

I feel really silly admitting this, but when I read this study I tried just pouring a lot of cold water into my ear and then quickly reviewing my opinions on controversial issues. Nothing unusual happened, so either the procedure requires a more complicated inner ear irrigation technique, or ear irrigation doesn't do anything special to non-anosognostic people, neither of which surprise me at all (guess it could also mean I'm just naturally right about everything :)

Comment author: Johnicholas 11 March 2009 05:17:14PM *  5 points [-]

A syllogism is three lines, each containing a quantifier, a subject, and a possibly negated predicate. It is a really rigid form of argument, and not tree-like at all. You may be thinking of a sorites, which is a bunch of syllogisms put together. Tree structured arguments are incredibly common in all kinds of logic, proof theory, and argumentation theory. Leaping from "tree-shaped" to "sorites" is like leaping from "flattish" to "flat-earthers".

Regardless of my nitpicking, I agree with you: we need progeny of these experiments. I may disagree about the details (predicate logic? lojban?!).

Comment author: Yvain 11 March 2009 10:23:35PM 1 point [-]

Thank you for the information on syllogisms. I know I was using the term wrong below, and I really should have known better. It may be nitpicking, but I think rationalists more than others are probably interested in making sure they use words correctly.

If you're familiar with Lojban, I'd be very interested in a post on how you think it would or wouldn't help with rationality.

The Apologist and the Revolutionary

159 Post author: Yvain 11 March 2009 09:39PM

Rationalists complain that most people are too willing to make excuses for their positions, and too unwilling to abandon those positions for ones that better fit the evidence. And most people really are pretty bad at this. But certain stroke victims called anosognosiacs are much, much worse.

Anosognosia is the condition of not being aware of your own disabilities. To be clear, we're not talking minor disabilities here, the sort that only show up during a comprehensive clinical exam. We're talking paralysis or even blindness1. Things that should be pretty hard to miss.

Take the example of the woman discussed in Lishman's Organic Psychiatry. After a right-hemisphere stroke, she lost movement in her left arm but continuously denied it. When the doctor asked her to move her arm, and she observed it not moving, she claimed that it wasn't actually her arm, it was her daughter's. Why was her daughter's arm attached to her shoulder? The patient claimed her daughter had been there in the bed with her all week. Why was her wedding ring on her daughter's hand? The patient said her daughter had borrowed it. Where was the patient's arm? The patient "turned her head and searched in a bemused way over her left shoulder".

Why won't these patients admit they're paralyzed, and what are the implications for neurotypical humans? Dr. Vilayanur Ramachandran, leading neuroscientist and current holder of the world land-speed record for hypothesis generation, has a theory.

One immediately plausible hypothesis: the patient is unable to cope psychologically with the possibility of being paralyzed, so he responds with denial. Plausible, but according to Dr. Ramachandran, wrong. He notes that patients with left-side strokes almost never suffer anosognosia, even though the left side controls the right half of the body in about the same way the right side controls the left half. There must be something special about the right hemisphere.

Another plausible hypothesis: the part of the brain responsible for thinking about the affected area was damaged in the stroke. Therefore, the patient has lost access to the area, so to speak. Dr. Ramachandran doesn't like this idea either. The lack of right-sided anosognosia in left-hemisphere stroke victims argues against it as well. But how can we disconfirm it?

Dr. Ramachandran performed an experiment2 where he "paralyzed" an anosognosiac's good right arm. He placed it in a clever system of mirrors that caused a research assistant's arm to look as if it was attached to the patient's shoulder. Ramachandran told the patient to move his own right arm, and the false arm didn't move. What happened? The patient claimed he could see the arm moving - a classic anosognosiac response. This suggests that the anosognosia is not specifically a deficit of the brain's left-arm monitoring system, but rather some sort of failure of rationality.

Says Dr. Ramachandran:

The reason anosognosia is so puzzling is that we have come to regard the 'intellect' as primarily propositional in character and one ordinarily expects propositional logic to be internally consistent. To listen to a patient deny ownership of her arm and yet, in the same breath, admit that it is attached to her shoulder is one of the most perplexing phenomena that one can encounter as a neurologist.

So what's Dr. Ramachandran's solution? He posits two different reasoning modules located in the two different hemispheres. The left brain tries to fit the data to the theory to preserve a coherent internal narrative and prevent a person from jumping back and forth between conclusions upon each new data point. It is primarily an apologist, there to explain why any experience is exactly what its own theory would have predicted. The right brain is the seat of the second virtue. When it's had enough of the left-brain's confabulating, it initiates a Kuhnian paradigm shift to a completely new narrative. Ramachandran describes it as "a left-wing revolutionary".

Normally these two systems work in balance. But if a stroke takes the revolutionary offline, the brain loses its ability to change its mind about anything significant. If your left arm was working before your stroke, the little voice that ought to tell you it might be time to reject the "left arm works fine" theory goes silent. The only one left is the poor apologist, who must tirelessly invent stranger and stranger excuses for why all the facts really fit the "left arm works fine" theory perfectly well.

It gets weirder. For some reason, squirting cold water into the left ear canal wakes up the revolutionary. Maybe the intense sensory input from an unexpected source makes the right hemisphere unusually aroused. Maybe distoring the balance sense causes the eyes to move rapidly, activating a latent system for inter-hemisphere co-ordination usually restricted to REM sleep3. In any case, a patient who has been denying paralysis for weeks or months will, upon having cold water placed in the ear, admit to paralysis, admit to having been paralyzed the past few weeks or months, and express bewilderment at having ever denied such an obvious fact. And then the effect wears off, and the patient not only denies the paralysis but denies ever having admitted to it.

This divorce between the apologist and the revolutionary might also explain some of the odd behavior of split-brain patients. Consider the following experiment: a split-brain patient was shown two images, one in each visual field. The left hemisphere received the image of a chicken claw, and the right hemisphere received the image of a snowed-in house. The patient was asked verbally to describe what he saw, activating the left (more verbal) hemisphere. The patient said he saw a chicken claw, as expected. Then the patient was asked to point with his left hand (controlled by the right hemisphere) to a picture related to the scene. Among the pictures available were a shovel and a chicken. He pointed to the shovel. So far, no crazier than what we've come to expect from neuroscience.

Now the doctor verbally asked the patient to describe why he just pointed to the shovel. The patient verbally (left hemisphere!) answered that he saw a chicken claw, and of course shovels are necessary to clean out chicken sheds, so he pointed to the shovel to indicate chickens. The apologist in the left-brain is helpless to do anything besides explain why the data fits its own theory, and its own theory is that whatever happened had something to do with chickens, dammit!

The logical follow-up experiment would be to ask the right hemisphere to explain the left hemisphere's actions. Unfortunately, the right hemisphere is either non-linguistic or as close as to make no difference. Whatever its thoughts, it's keeping them to itself.

...you know, my mouth is still agape at that whole cold-water-in-the-ear trick. I have this fantasy of gathering all the leading creationists together and squirting ice cold water in each of their left ears. All of a sudden, one and all, they admit their mistakes, and express bafflement at ever having believed such nonsense. And then ten minutes later the effect wears off, and they're all back to talking about irreducible complexity or whatever. I don't mind. I've already run off to upload the video to YouTube.

This is surely so great an exaggeration of Dr. Ramachandran's theory as to be a parody of it. And in any case I don't know how much to believe all this about different reasoning modules, or how closely the intuitive understanding of it I take from his paper matches the way a neuroscientist would think of it. Are the apologist and the revolutionary active in normal thought? Do anosognosiacs demonstrate the same pathological inability to change their mind on issues other than their disabilities? What of the argument that confabulation is a rather common failure mode of the brain, shared by some conditions that have little to do with right-hemisphere failure? Why does the effect of the cold water wear off so quickly? I've yet to see any really satisfying answers to any of these questions.

But whether Ramachandran is right or wrong, I give him enormous credit for doing serious research into the neural correlates of human rationality. I can think of few other fields that offer so many potential benefits.

 

Footnotes

1: See Anton-Babinski syndrome

2: See Ramachandran's "The Evolutionary Biology of Self-Deception", the link from "posits two different reasoning modules" in this article.

3: For Ramachandran's thoughts on REM, again see "The Evolutionary Biology of Self Deception"

Comment author: Psy-Kosh 11 March 2009 07:01:54AM 1 point [-]

Hrm... you seem to have better google-fu than I. So, what is "evidential decision theory" then? or at least, can you point me to whichever link you found that actually had basic description on it? Thanks.

Comment author: Yvain 11 March 2009 08:30:42AM 1 point [-]

Nick Tarleton's "Smoking Lesion" link just below is the best introduction I've found so far.

Comment author: Johnicholas 10 March 2009 05:51:07PM 3 points [-]

If I am not mistaken, you have several criticisms of truthmapper. I've tried to respond to them in a carefully numbered fashion. This separation might be a rough approximation of the way a software tool would structure an argument.

  1. A proof from the sole premise 'A is A' concluding 'Taxation is slavery' is certainly fallacious, I agree. Can you expand on what the 'same reason' is? I'm not sure what I'm expected to see in the argument you reference. It is awkward at the very least, but it is more detailed, concrete and falsifiable than many trollish claims, and some of its flaws are pointed out in the critiques.

  2. The site may encourage people to be sloppy in their argumentation, or it may make sloppiness more obvious.

  3. The video makes a fallacious claim "all assumptions explicit", and that diminishes my trust of the organization, I agree.

  4. I'm not sure what you mean by "argument should work like an Aristotelian syllogism". There are many flaws in syllogisms - the one I remember is the inability to prove that a horse's head is an animal's head. Because the claims are natural language text, the structure truthmapper enforces is looser than a syllogism; merely a tree of claims and supporting claims.

  5. You're entirely correct, the logo is not good.

  6. Paying official attention to argumentation may encourage making it a status contest, with individuals striving to "win" rather than striving to discover the truth. This is a thorny problem for rationality, but I don't think it is confined to argumentation software.

Comment author: Yvain 11 March 2009 12:29:11AM *  4 points [-]

I think we only disagree on 4. (You agree with me on 2,3, and 5, and I agree with you that 6 is not confined to software). I think the expansion of 1 you want really is 4, and I admit I explained 4 poorly. It is kind of tangled in my own head, but maybe I can do better:

TruthMapper encourages people to think that an argument on politics or religion or culture is structured like a deductive proof, where if there's a problem, it's because someone accidentally used (A -> B) and (B) to conclude (A) or something silly like that. The real problem with all of these arguments is that no one's grounded their morality properly, people are treating generalizations as universals, people import hidden assumptions, people think proving a single major negative of a disliked theory is enough without running a cost-benefit analysis, people are using words wrongly and so on.

But upon further thought, you're right that this is the program making a common flaw more obvious, not the program creating the flaw. But if I were to encounter for example the argument about art on a message board, I would try to explain why the whole argument was hopeless because of these points, and how the person's argument style could become more rigorous. Whereas on TruthMapper, I am reduced to sniping at why Point 4 doesn't follow from Point 3.

But I'm open to testing the system empirically. I trust the people here to avoid the sort of mistakes the people in the art argument used. If you want to organize a LessWrong debate about something on TruthMapper, I'll participate and change my mind if the debate goes better than it would on a comment thread here.

In response to Striving to Accept
Comment author: Yvain 10 March 2009 04:44:04PM 7 points [-]

I hadn't thought about a normative aspect to the skeptics' fear of ghosts, but after reading this post I acknowledge that there is one. I commit to fighting irrational fears from now on.

I read with interest all of the specific techniques proposed on this thread and welcome more. I'd be particularly interested if anyone had theories of or techniques against akrasia, the general inability of the mind to follow through on what it knows to be right.

Comment author: Yvain 10 March 2009 04:28:22PM *  5 points [-]

TruthMapper scares me, for the same reason Objectivists I used to know who thought they knew a formal deductive proof going from "A is A" to "Taxation is slavery", justifying each step with an inference rule scared me.

See for example the proof that commercialization of fine art hurts society.

I'm not sure whether TruthMapper encourages people to be sloppy, or whether it's such a good tool that the sloppiness is just much more obvious than it would be on a message board.

But I'm inclined to lay a bit of the blame on the site itself. For one thing, the video claims that it lets people make all assumptions explicit, which I take to mean that the company behind it believes that. For another, the entire philosophy seems to be that argument should work like an Aristotelian syllogism, and that's part of the problem. For a third, I can't take them seriously with that logo. Did they pay the designer per Photoshop layer effect used?

Debategraph looks like a mind map kind of thing. I suppose if that's the way you like seeing your information organized, it could be useful. I'm just wary of the whole concept of formalizing debate too much (by formal, I mean formal as in official, not formal as in formal systems). Once you start thinking like a high school kid at Debate Club, you've already lost, and I worry these sites could encourage that mode.

The idea of truth-seeking software is a good one, but there's got to be a way to avoid aiming it at the lowest common denominator.

Comment author: Nick_Tarleton 10 March 2009 12:48:55AM 4 points [-]

If God models your actions and pre-decides to save or damn you accordingly, sure. But if, as you say, behavior is caused by elect status, it seems to me that's exactly the same as the Smoking Lesion (where CDT is in the right).

Comment author: Yvain 10 March 2009 02:51:15AM 3 points [-]

I see a difference between the two cases as follows:

The Smoking Lesion, as I interpret it when I say it seems to be self-evidently a correct counterexample to EDT, only affects your craving to smoke. Your probability of getting lung cancer is therefore affected only by your craving to smoke. Once you note whether or not you have a strong craving for cigarettes, your choice whether or not to smoke provides no further evidence for your possession of the lesion than the craving alone did.

As I interpret Calvinism, your status as elect or reprobate doesn't just cause a craving to live virtuously. It actually causes the entire decision. Unlike in the smoking lesion, my internal argument about which decision theory I use to decide the Calvinism problem is itself part of the evidence for whether I am among the elect or not. This gives trying to work out a decision theory about Calvinism a self-referential element that the smoking lesion completely lacks.

As I said to Eliezer, I would be much more confident that living a virtuous life is the correct decision if I noted that many Calvinists before me had engaged in this exact argument about decision theories, and every single one who eventually came to hold my opinion went to Heaven and every single one who rejected it went to Hell.

If we used the same constraint as in the Augustine's Paradox thread - that God is interested in your decision-making algorithm and rejects anyone who lives a virtuous life for reasons other than their native disposition, then I agree with you and I would live a life of sin.

Comment author: Eliezer_Yudkowsky 10 March 2009 12:35:04AM 3 points [-]

I still choose to be a virtuous Calvinist

"Choose"?

Comment author: Yvain 10 March 2009 01:15:27AM *  7 points [-]

Yes, choose!

I don't find Cleonid's comparison with run-of-the-mill determinism inappropriate. The position of particles during the infancy of the universe completely determined whether or not I will get out of bed tomorrow morning. In that sense, I am a get-out-of-bed Calvinist: my fate was already decided at the moment of Creation, and I am either one of the elect whose genes/environment/etc predispose him to rising, or one of the reprobate whose genes/environment/etc doom him to stay in bed all day.

And yet I wake up tomorrow morning, and I find myself presented with what seems like a choice to get out of bed or not.

I am happy to say I "choose" to get out of bed, and I am happy to say I would "choose" to be a virtuous Calvinist. Even though I'm imbuing the word "choose" with less mystical force than a believer in free will might.

(here I am assuming a relationship between spiritual causes and the soul that's similar to the relationship between physical causes and the brain you explained in Thou Art Physics. As far as I know, Calvinists believe that God's choice is mediated through a change in the effect of original sin on the soul of the person He has chosen, which seems to fit the bill.)

How can I make this clearer...okay. Let's say there have been a trillion Calvinists throughout history. They've all been rationalists and they've all engaged in this same argument. Some of them have been pro-sin for the same reasons you are, others have been pro-virtue for the same reasons I am. Some on each side have changed their minds after having listened to the arguments. And of all of these trillion Calvinists, every single one who after all the arguments decides to live a life of virtue - has gone to Heaven. And every single one who, after all the arguments, decides to live a life of sin - has gone to Hell.

To say that you have no reason to change your mind here seems to be suggesting that there's a pretty good chance you will be the exception to a rule that has held 100% of the time in previous cases: the sinful Calvinist who goes to Heaven, or the virtuous Calvinist who goes to Hell. If this never worked for a trillion people in your exact position, why do you think it will work for you now?

Comment author: Yvain 10 March 2009 12:55:01AM *  10 points [-]

I'll bite the bullet. Your reasoning is correct. Assuming the laws are just and everything scales linearly, we should change our legal system to increase penalties for rarely-detected crimes.

The "everything scales linearly" is a big assumption, though. It's not very clear that deterrence scales linearly: I don't remember whether the RIAA's last suit for illegal music downloads made them ten thousand or ten million dollars, but I don't think I'd be a thousand times less likely (or even slightly less likely) to download music in the second case. If there's no difference between the deterrence value of the two cases, then society just took $9990000 worth of utility away from someone for zero gain.

And punishment doesn't scale exactly linearly either: fining 100% of a person's net worth is more than twice as bad as fining 50% of a person's net worth.

So I'm not sure I would be willing to fully endorse the new punishment scheme unless social scientists and statisticians came up with a model that adjusted for all of this in order to produce the greatest net happiness to society. But in a hypothetical situation where there was a perfect statistical model for all of this...go for it.

Also, great title :)

Comment author: Eliezer_Yudkowsky 09 March 2009 11:25:43PM 2 points [-]

See also "evidential decision theory", the wrong reason to one-box on Newcomb's Problem.

Comment author: Yvain 10 March 2009 12:29:53AM *  6 points [-]

Thanks for the term. I've Googled it and had a very brief look over evidential versus causal decision theory (will look more later). And I accept that the evidential theory is flawed, and the causal theory is better, and that EDT gives the wrong result on the smoking problem, and so on.

But keeping the original premise that it's known that out of everyone who's ever lived in all of history, every single virtuous Calvinist has ended up in Heaven and every single sinful Calvinist end has ended up damned - I still choose to be a virtuous Calvinist. And if the decision theorists don't like that, they can go to hell.

You May Already Be A Sinner

41 Post author: Yvain 09 March 2009 11:18PM

Followup to: Simultaneously Right and Wrong

Related to: Augustine's Paradox of Optimal Repentance

"When they inquire into predestination, they are penetrating the sacred precincts of divine wisdom. If anyone with carefree assurance breaks into this place, he will not succeed in satisfying his curiosity and he will enter a labyrinth from which he can find no exit."

            -- John Calvin

John Calvin preached the doctrine of predestination: that God irreversibly decreed each man's eternal fate at the moment of Creation. Calvinists separate mankind into two groups: the elect, whom God predestined for Heaven, and the reprobate, whom God predestined for eternal punishment in Hell.

If you had the bad luck to be born a sinner, there is nothing you can do about it. You are too corrupted by original sin to even have the slightest urge to seek out the true faith. Conversely, if you were born one of the elect, you've got it pretty good; no matter what your actions on Earth, it is impossible for God to revoke your birthright to eternal bliss.

However, it is believed that the elect always live pious, virtuous lives full of faith and hard work. Also, the reprobate always commit heinous sins like greed and sloth and commenting on anti-theist blogs. This isn't what causes God to damn them. It's just what happens to them after they've been damned: their soul has no connection with God and so it tends in the opposite direction.

Consider two Calvinists, Aaron and Zachary, both interested only in maximizing his own happiness. Aaron thinks to himself "Whether or not I go to Heaven has already been decided, regardless of my actions on Earth. Therefore, I might as well try to have as much fun as possible, knowing it won't effect the afterlife either way." He spends his days in sex, debauchery, and anti-theist blog comments.

Zachary sees Aaron and thinks "That sinful man is thus proven one of the reprobate, and damned to Hell. I will avoid his fate by living a pious life." Zachary becomes a great minister, famous for his virtue, and when he dies his entire congregation concludes he must have been one of the elect.

Before the cut: If you were a Calvinist, which path would you take?

Amos Tversky, Stanford psychology professor by day, bias-fighting superhero by night, thinks you should live a life of sin. He bases his analysis of the issue on the famous maxim that correlation is not causation. Your virtue during life is correlated to your eternal reward, but only because they're both correlated to a hidden third variable, your status as one of the elect, which causes both.

Just to make that more concrete: people who own more cars live longer. Why? Rich people buy more cars, and rich people have higher life expectancies. Both cars and long life are caused by a hidden third variable, wealth. Trying to increase your chances of getting into Heaven by being virtuous is as futile as trying to increase your life expectancy by buying another car.

Some people would stop there, but not Amos Tversky, bias-fighting superhero. He and George Quattrone conducted a study that both illuminated a flaw in human reasoning about causation and demonstrated yet another way people can be simultaneously right and wrong.

Subjects came in thinking it was a study on cardiovascular health. First, experimenters tested their pain tolerance by making them stick their hands in a bucket of freezing water until they couldn't bear it any longer. However long they kept it there was their baseline pain tolerance score.

Then experimenters described two supposed types of human heart: Type I hearts, which work poorly and are prone to heart attack and will kill you at a young age, and Type II hearts, which work well and will bless you with a long life. You can tell a Type I heart from a Type II heart because...and here the subjects split into two groups. Group A learned that people with Type II hearts, the good hearts, had higher pain tolerance after exercise. Group B learned that Type II hearts had lower pain tolerance after exercise.

Then the subjects exercised for a while and stuck their hands in the bucket of ice water again. Sure enough, the subjects who thought increased pain tolerance meant a healthier heart kept their hands in longer. And then when the researchers went and asked them, they said they must have a Type II heart because the ice water test went so well!

The subjects seem to have believed on some level that keeping their hand in the water longer could give them a different kind of heart. Dr. Tversky declared that people have a cognitive blind spot to "hidden variable" causation, and this explains the Calvinists who made such an effort to live virtuously.

But this study is also interesting as an example of self-deception. One level of the mind made the (irrational) choice to leave the hand in the ice water longer. Another level of the mind that wasn't consciously aware of this choice interpreted it as evidence for the Type II heart. There are two cognitive flaws here: the subject's choice to try harder on the ice water test, and his lack of realization that he'd done so.

I don't know of any literature explicitly connecting this study to self-handicapping, but the surface similarities are striking. In both, a person takes an action intended to protect his self-image that will work if and only if he doesn't realize this intent. In both, the action is apparently successful, self-image is protected, and the conscious mind remains unaware of the true motives.

Despite all this, and with all due respect to Dr. Tversky I think he might be wrong about the whole predestination issue. If I were a Calvinist, I'd live a life of sin if and only if I would two-box on Newcomb's Problem.

Comment author: Yvain 09 March 2009 08:04:52PM 1 point [-]

Thank you. Trying it now and it's working perfectly. It's like a dream come true.

Comment author: roland 08 March 2009 11:53:18PM 67 points [-]

The reasoning mistake that Yvain and a lot of people here are making is: they think that if someone is scared in a supposedly haunted house there must be a believe in ghosts hidden somewhere inside his brain. What happens in reality is that the mind is hardwired to be scared when certain conditions are met. Being out in the dark is scary, not because you have a believe in ghosts but because there used to be predators roaming about in the ancestral environment and so the brain triggers accordingly. Now this whole ghost issue is probably a post-factum rationalization. Our verbal reasoning just pops out with an explanation of why we are scared(I was scared without a reason so I must have a believe in ghosts somewhere in my mind!). The real reason is below the surface and inacessible because we lack the ability for introspection.

Comment author: Yvain 09 March 2009 12:01:47AM 22 points [-]

I think....I agree with you.

Not totally, because the source of the fear can't be entirely evolutionary - I could be not the least bit afraid of the dark in normal life but become very afraid when in a supposedly haunted area. But I think your general point that the thoughts about ghosts are a constructed explanation for the fear rather than the source of the fear is the basis for a solution to the mystery.

Maybe there's a vicious cycle, where thinking about ghosts increases the salience of darkness and mysterious noises that normally wouldn't trouble you, you become afraid, the fear increases the salience of ghosts, and so on.

Comment author: roland 08 March 2009 11:27:14PM *  10 points [-]

In the normal hotel, I would shrug off any unusual noises. In the old haunted mansion, I would run out screaming. The difference between them can best be explained by the mansion's ghostly associations.

Sure, because there are other people around in the hotel. You probably would feel the same if you spent the night in the haunted mansion if it's full of people. Now take that hotel, put it in a remote location and without anybody except yourself. Scary, right?

Going alone into a dark forest at night is scary, going with a group of people can be quite fun.

Comment author: Yvain 08 March 2009 11:33:02PM 3 points [-]

Maybe this is a case of honestly different intuitions. My intuition is that if we control for everything except for one building having a reputation for being haunted and the other not, I would still be more scared in the haunted building.

Comment author: roland 08 March 2009 09:37:01PM 7 points [-]

So although it's correct to say that the skeptics' emotions overwhelmed their rationality, they wouldn't have those emotions unless they thought on some level that ghosts were worth getting scared about.

I disagree. We have the emotions because our brain is designed to react with alarm to nearby strange sounds. It has nothing to do with ghosts. It's simply the primitive brain circuitry firing.

Regarding the fan did you read this section? http://en.wikipedia.org/wiki/Fan_death#Fan-related_hyperthermia_research It seems that in fact a fan can cause death, although not by the reasons believed. So indeed, sleeping with a fan on a very hot night could prove fatal.

Comment author: Yvain 08 March 2009 10:14:05PM 9 points [-]

I disagree. Compare spending the night in a normal hotel with spending the night in an old mansion that is widely believed to be haunted.

In the normal hotel, I would shrug off any unusual noises. In the old haunted mansion, I would run out screaming. The difference between them can best be explained by the mansion's ghostly associations.

I did see that in the article about fan death, but there are a lot of things that can cause very occasional deaths in very specific circumstances. Compare this to a Korean I used to know, for whom sleeping with a fan would be literally a terrifying experience.

The Mystery of the Haunted Rationalist

69 Post author: Yvain 08 March 2009 08:39PM

Followup to: Simultaneously Right and Wrong

    "The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents."

          - H.P. Lovecraft, The Call of Cthulhu

There is an old yarn about two skeptics who stayed overnight in a supposedly haunted mansion, just to prove they weren't superstitious. At first, they laughed and joked with each other in the well-lit master bedroom. But around eleven, there was a thunderstorm - hardly a rare event in those parts - and all the lights went off. As it got later and later, the skeptics grew more and more nervous, until finally around midnight, the stairs leading up to their room started to creak. The two of them shot out of there and didn't stop running until they were in their car and driving away.

So the skeptics' emotions overwhelmed their rationality. That happens all the time. Is there any reason to think this story proves anything more interesting than that some skeptics are cowards?

The Koreans have a superstition called "fan death": if you sleep in a closed room with a fan on all night, you will die. Something about the fan blades shredding the oxygen molecules or something. It all sounds pretty far-fetched, but in Korea it's endorsed by everyone from doctors to the government's official consumer safety board.

I don't believe in ghosts, and I don't believe in fan death. But my reactions to spending the night in a haunted mansion and spending the night with a fan are completely different. Put me in a haunted mansion, and I'll probably run out screaming the first time something goes bump in the night1. Put me in a closed room with a fan and I'll shrug and sleep like a baby. Not because my superior rationality has conquered my fear. Because fans just plain don't kill people by chopping up oxygen, and to think otherwise is just stupid.

So although it's correct to say that the skeptics' emotions overwhelmed their rationality, they wouldn't have those emotions unless they thought on some level that ghosts were worth getting scared about.

A psychologist armed with the theory of belief-profession versus anticipation-control would conclude that I profess disbelief in ghosts to fit in with my rationalist friends, but that I anticipate being killed by a ghost if I remain in the haunted mansion. He'd dismiss my skepticism about ghosts as exactly the same sort of belief in belief afflicting the man who thinks his dragon is permeable to flour.

If this psychologist were really interested in investigating my beliefs, he might offer me X dollars to stay in the haunted mansion. This is all a thought experiment, so I can't say for certain what I would do. But when I imagine the scenario, I visualize myself still running away when X = 10, but fighting my fear and staying around when X = 1000000.

This looks suspiciously like I'm making an expected utility calculation. Probability of being killed by ghost * value of my life, compared to a million dollars. It also looks like I'm using a rather high number for (probability of being killed by ghost): certainly still less than .5, but much greater than the <.001 I would consciously assign it. Is my mind haunted by an invisible probability of ghosts, ready to jump out and terrify me into making irrational decisions?

How can I defend myself against the psychologist's accusation that I merely profess a disbelief in ghosts? Well, while I am running in terror out of the mansion, a bookie runs up beside me. He offers me a bet: he will go back in and check to see if there is a ghost. If there isn't, he owes me $100. If there is, I owe him $10,000 (payable to his next of kin). Do I take the bet?

Thought experiments don't always work, but I imagine myself taking the bet. I assign a less than 1/100 chance to the existence of ghosts, so it's probably a good deal. The fact that I am running away from a ghost as I do the calculation changes the odds not at all.

But if that's true, we're now up to three different levels of belief. The one I profess to my friends, the one that controls my anticipation, and the one that influences my emotions.

There are no ghosts, profess skepticism.
There are no ghosts, take the bet.
There are ghosts, run for your life!

Footnote

1: I worry when writing this that I may be alone among Less Wrong community members, and that the rest of the community would remain in the mansion with minimal discomfort. If "run screaming out of the mansion" is too dramatic for you, will you agree that you might, after the floorboards get especially creaky, feel a tiny urge to light a candle or turn on a flashlight? Even that is enough to preserve the point I am trying to make here.

Simultaneously Right and Wrong

88 Post author: Yvain 07 March 2009 10:55PM

Related to: Belief in Belief, Convenient Overconfidence

     "You've no idea of what a poor opinion I have of myself, and how little I deserve it."

      -- W.S. Gilbert

In 1978, Steven Berglas and Edward Jones performed a study on voluntary use of performance inhibiting drugs. They asked subjects to solve certain problems. The control group received simple problems, the experimental group impossible problems. The researchers then told all subjects they'd solved the problems successfully, leaving the controls confident in their own abilities and the experimental group privately aware they'd just made a very lucky guess.

Then they offered the subjects a choice of two drugs to test. One drug supposedly enhanced performance, the other supposedly handicapped it.

There's a cut here in case you want to predict what happened.

Males in the lucky guesser group chose the performance-inhibiting drug significantly more than those in the control group1.

The researchers conjectured that the lucky guessers felt good about acing the first test. They anticipated failing the second whether they chose the helpful or the harmful drug. So they chose the harmful drug as an excuse: "Oh, I would have passed the test, only the drug was making me stupid." As the study points out, this is a win-win situation: if they fail, the drug excuses their failure, and if they succeed it's doubly impressive that they passed even with a handicap.

Since this study, psychologists have applied self-handicapping to interpret common phenomena like procrastination, hypochondria, and minority underachievement; they've also discovered factors that increase or decrease self-handicapping tendencies (for example, you can increase it by making the test more relevant to your self-esteem: just say "this is a proven test of intellectual ability in general").

But some especially interesting studies investigated the effect of privacy on self-handicapping. For example, Hobden conducted an experiment similar to Berglas and Jones', albeit with tapes of performance-enhancing or -handicapping music instead of drugs. The twist: half the subjects' choice of tapes and test scores were public, and half the subjects believed their choice of tapes and test scores were anonymous. What happens when no one but the subject himself will ever know his test score? He self-handicaps just as often as everyone else. And it seems to *work*. The same set of studies showed that subjects who self-handicap on a test are less likely to attribute their failure on the test to their own incompetence.

In order to handicap, subjects must have an inaccurate assessment of their own abilities. Otherwise, there's no self-esteem to protect. If I believe my IQ is 80, and I get 80 on an IQ test, I have no incentive to make excuses to myself, or to try to explain away the results. The only time I would want to explain away the results as based on some external factor was if I'd been going around thinking my real IQ was 100.

But subjects also must have an accurate assessment of their own abilities. Subjects who take an easy pre-test and expect an easy test do not self-handicap. Only subjects who understand their low chances of success can think "I will probably fail this test, so I will need an excuse2.

If this sounds familiar, it's because it's another form of the dragon problem from Belief in Belief. The believer says there is a dragon in his garage, but expects all attempts to detect the dragon's presence to fail. Eliezer writes: "The claimant must have an accurate model of the situation somewhere in his mind, because he can anticipate, in advance, exactly which experimental results he'll need to excuse."

Should we say that the subject believes he will get an 80, but believes in believing that he will get a 100? This doesn't quite capture the spirit of the situation. Classic belief in belief seems to involve value judgments and complex belief systems, but self-handicapping seems more like simple overconfidence bias3. Is there any other evidence that overconfidence has a belief-in-belief aspect to it?

Last November, Robin described a study where subjects were less overconfident if asked to predict their performance on tasks they will actually be expected to complete. He ended by noting that "It is almost as if we at some level realize that our overconfidence is unrealistic."

Belief in belief in religious faith and self-confidence seem to be two areas in which we can be simultaneously right and wrong: expressing a biased position on a superficial level while holding an accurate position on a deeper level. The specifics are different in each case, but perhaps the same general mechanism may underlie both. How many other biases use this same mechanism?

Footnotes

1: In most studies on this effect, it's most commonly observed among males. The reasons are too complicated and controversial to be discussed in this post, but are left as an exercise for the reader with a background in evolutionary psychology.

2: Compare the ideal Bayesian, for whom expected future expectation is always the same as the current expectation, and investors in an ideal stock market, who must always expect a stock's price tomorrow to be on average the same as its price today - to this poor creature, who accurately predicts that he will lower his estimate of his intelligence after taking the test, but who doesn't use that prediction to change his pre-test estimates.

3: I have seen "overconfidence bias" used in two different ways: to mean poor calibration on guesses (ie predictions made with 99% certainty that are only right 70% of the time) and to mean the tendency to overestimate one's own good qualities and chance of success. I am using the latter definition here to remain consistent with the common usage on Overcoming Bias; other people may call this same error "optimism bias".

Comment author: CronoDAS 08 March 2009 04:25:09AM *  6 points [-]

Yes, it's worth another post - I hadn't heard that theory before.

::runs off to do some Google searches::

Some difficult work with Google revealed that the technical term is the "sociometer" theory - and it's fairly recent (the oldest citation I see refers to 1995), which would help explain why I hadn't heard of it before. It seems consistent with my personal experiences, so I consider it credible.

For more information:

http://www.psychwiki.com/wiki/Sociometer_Theory

Comment author: Yvain 08 March 2009 08:15:45PM 4 points [-]

Okay, I'll definitely post on sociometer theory sometime.

In response to Lies and Secrets
Comment author: Yvain 08 March 2009 06:58:29PM 14 points [-]

I think we inherit this idea of "lies are worse than secrets" from classic deontological morality (an act-utilitarian could try to quantify the harm caused by each and compare, so he doesn't have as deep a problem)

In my opinion, a lot of deontological morality is rooted in a system for minimizing blame The guy who wouldn't push the fat man onto the tracks to save the people in the trolley problem ( http://en.wikipedia.org/wiki/Trolley_problem#The_fat_man ) knows that he couldn't be blamed for the trolley victims' death, but he could and would be blamed for the murder of the fat man. Therefore, he concludes that leaving the people on the track to die is more moral. Now, looking at your analogy:

"If a man is starving, not giving him a loaf of bread is as deadly as giving him cyanide. But if there are a lot of random objects lying around in the neighborhood, the former deed is less deadly: it's far more likely that one of the random objects is a loaf of bread than that it is an antidote to cyanide."

If I give a man cyanide, then I am clearly and visibly to blame for his death. From a status point of view, that's social suicide. If I fail to give him a loaf of bread, society isn't going to come knocking down my door to drag me to jail. For one thing, no one will even associate his death with me. For another, everyone else who didn't give him bread will be equally to blame. Therefore in classical morality, giving him cyanide is an evil act and not giving him bread is a neutral or barely-evil act.

Now apply that to the lies versus secrets question. If I tell a lie, and I'm caught, then people have every right to get mad at me. If I don't tell you some information that is necessary for you to succeed, you can never prove that I did it intentionally and everyone else who failed to give you that information is equally guilty. Therefore classical morality considers lying worse than keeping secrets, even in cases where utilitarian morality says I've harmed you exactly the same amount either way.

If we alter the situation to make it easier to pin the blame on me, classical morality starts condemning me more. If I am the sole witness in an important criminal case and I describe the entire scene accurately except that I fail to mention I saw the suspect there with a knife, and then a videotape later shows me at the scene, staring at the suspect, I will get in trouble. In this case, it's easy to blame me for not conveying the information, since I was ritually placed in a position where conveying the information was my sole responsibility and since it can be proven I withheld the information intentionally. And in this situation, most people would consider my withholding of information "immoral".

Comment author: Yvain 08 March 2009 11:34:35AM 3 points [-]

Converting between time and money sounds like a good solution to this problem.

If you're interested in the broader psychological mechanism behind this bias, I highly recommend Jonah Lehrer' s blog post on irrational purchasing decisions: http://scienceblogs.com/cortex/2009/02/shopping.php

Comment author: Cameron_Taylor 08 March 2009 04:37:40AM 1 point [-]

The ideal Bayesian [can] never predict in which direction future information will alter his own estimates, and investors in an ideal stock market, [can] never predict in which direction prices will move

I suggest rewording this, it seems like you are making a different claim than the one you intended. An ideal Bayesian can predict in which direction future information will alter his own estimates.

I have been given a coin which I know is either fair or biased (comes up heads 75% of the time). After a sequence of tosses I have arrived at, say, 95% probability that the coin is biased. The probability I assign to the next toss giving 'heads' is:

p(heads) = 0.95 * 0.75 + 0.05 * 0.5 ~= 0.74

There is a 74% chance that I will alter my estimate upwards after this coin toss.

I predict with 95% confidence that should I continue to toss the coin long enough future information will alter my estimates upwads until it reaches ~100% confidence that the coin is biased. Naturally, I predict a 5% chance that my estimates would be eventually altered downwards until they approximate 0%, a far greater change.

The same applies to some stocks in an ideal stock market. For example, some companies may have a limit on their growth potential and yet have some chance of going bankrupt. The chance that these stocks could completley lose value should suggest that they are more likely to go up than to go down for their value to be what it is now.

Can someone suggest a concise replacement for "in which direction" that applies here?

Comment author: Yvain 08 March 2009 11:10:20AM 2 points [-]

You're right. Edited to Jim's version, although it sounds kind of convoluted. I'm going to keep an eye out for how real statisticians describe this.

Comment author: infotropism 08 March 2009 12:00:28AM 4 points [-]

Looks like it's related to learned helplessness to me.

http://en.wikipedia.org/wiki/Learned_helplessness

Comment author: Yvain 08 March 2009 01:53:36AM 7 points [-]

The relationship discussed in the literature mostly involves them as two competing explanations for underachievement. Learned helplessness is about internalizing the conception of yourself as worthless; self-handicapping is about trying as hard as you can to avoid viewing yourself as worthless. The studies I could find in ten minutes on Google Scholar mostly suggested a current consensus that run-of-the-mill underachievers are sometimes self-handicappers but not learned-helplessness victims - but ten minutes does not a literature review make.

Oh, and thank you for linking to that Wikipedia article. The sentence about how "people performed mental tasks in the presence of distracting noise...if the person could use a switch to turn off the noise, his performance improved, even though he rarely bothered to turn off the noise. Simply being aware of this option was enough to substantially counteract its distracting effect" is really, really interesting.

Comment author: Eliezer_Yudkowsky 07 March 2009 11:51:29PM 7 points [-]

Excellent post - it makes me wish that the system gave out a limited number of super-votes, like 1 for every 20 karma, so that I could vote this up twice.

I hope you don't mind, but I did a quick edit to insert "a choice of" before "two drugs to test", because that wasn't clear on my first reading. (Feel free to revert if you prefer your original wording.) Also edited the self-deception tag to self_deception per previous standard.

Comment author: Yvain 08 March 2009 01:28:28AM 9 points [-]

Thank you. Since I learned practically everything I know about rationality either from you or from books you recommended, I'm very happy to earn your approval...but also a little amused, since I consciously tried to copy your writing style as much as I could without actually inserting litanies.

Comment author: GuySrinivasan 08 March 2009 12:18:04AM 13 points [-]

My first reaction is that the 80-IQ guy needs to carry around a mental model of himself as a 100-IQ guy for status purposes, and a mental model of himself as an 80-IQ guy for accuracy purposes. Possibly neither consciously.

(Is this availability bias at work because I have recently read lots of Robin's etc. writings on status?)

If true, I don't think there's any need to say he "believes" his IQ is 100 when it is in fact 80. We could just say he has at least one public persona which he'd like to signal has an IQ of 100, and that sometimes he draws predictions using this model rather than a more correct one, like when he's guaranteed privacy.

Comment author: Yvain 08 March 2009 01:10:57AM 13 points [-]

I agree with your first paragraph, but I don't quite understand your second.

In particular, I don't understand what you mean by there being no need to say he "believes". If upon being asked he would assert that his IQ is 100, and he wouldn't be consciously aware of lying, isn't that enough to say he believes his IQ is 100 on at least one level?

(also, when I say I agree with your first paragraph, I do so on the assumption that we mean the same thing by status. In particular, I would describe the "status" in this case as closer to "self-esteem" than "real position in a social hierarchy". Are most Less Wrong readers already aware of the theory that self-esteem is the way the calculation of status feels from the inside, or is that worth another post?)

Simultaneously Right and Wrong

88 Post author: Yvain 07 March 2009 10:55PM

Related to: Belief in Belief, Convenient Overconfidence

     "You've no idea of what a poor opinion I have of myself, and how little I deserve it."

      -- W.S. Gilbert

In 1978, Steven Berglas and Edward Jones performed a study on voluntary use of performance inhibiting drugs. They asked subjects to solve certain problems. The control group received simple problems, the experimental group impossible problems. The researchers then told all subjects they'd solved the problems successfully, leaving the controls confident in their own abilities and the experimental group privately aware they'd just made a very lucky guess.

Then they offered the subjects a choice of two drugs to test. One drug supposedly enhanced performance, the other supposedly handicapped it.

There's a cut here in case you want to predict what happened.

Males in the lucky guesser group chose the performance-inhibiting drug significantly more than those in the control group1.

The researchers conjectured that the lucky guessers felt good about acing the first test. They anticipated failing the second whether they chose the helpful or the harmful drug. So they chose the harmful drug as an excuse: "Oh, I would have passed the test, only the drug was making me stupid." As the study points out, this is a win-win situation: if they fail, the drug excuses their failure, and if they succeed it's doubly impressive that they passed even with a handicap.

Since this study, psychologists have applied self-handicapping to interpret common phenomena like procrastination, hypochondria, and minority underachievement; they've also discovered factors that increase or decrease self-handicapping tendencies (for example, you can increase it by making the test more relevant to your self-esteem: just say "this is a proven test of intellectual ability in general").

But some especially interesting studies investigated the effect of privacy on self-handicapping. For example, Hobden conducted an experiment similar to Berglas and Jones', albeit with tapes of performance-enhancing or -handicapping music instead of drugs. The twist: half the subjects' choice of tapes and test scores were public, and half the subjects believed their choice of tapes and test scores were anonymous. What happens when no one but the subject himself will ever know his test score? He self-handicaps just as often as everyone else. And it seems to *work*. The same set of studies showed that subjects who self-handicap on a test are less likely to attribute their failure on the test to their own incompetence.

In order to handicap, subjects must have an inaccurate assessment of their own abilities. Otherwise, there's no self-esteem to protect. If I believe my IQ is 80, and I get 80 on an IQ test, I have no incentive to make excuses to myself, or to try to explain away the results. The only time I would want to explain away the results as based on some external factor was if I'd been going around thinking my real IQ was 100.

But subjects also must have an accurate assessment of their own abilities. Subjects who take an easy pre-test and expect an easy test do not self-handicap. Only subjects who understand their low chances of success can think "I will probably fail this test, so I will need an excuse2.

If this sounds familiar, it's because it's another form of the dragon problem from Belief in Belief. The believer says there is a dragon in his garage, but expects all attempts to detect the dragon's presence to fail. Eliezer writes: "The claimant must have an accurate model of the situation somewhere in his mind, because he can anticipate, in advance, exactly which experimental results he'll need to excuse."

Should we say that the subject believes he will get an 80, but believes in believing that he will get a 100? This doesn't quite capture the spirit of the situation. Classic belief in belief seems to involve value judgments and complex belief systems, but self-handicapping seems more like simple overconfidence bias3. Is there any other evidence that overconfidence has a belief-in-belief aspect to it?

Last November, Robin described a study where subjects were less overconfident if asked to predict their performance on tasks they will actually be expected to complete. He ended by noting that "It is almost as if we at some level realize that our overconfidence is unrealistic."

Belief in belief in religious faith and self-confidence seem to be two areas in which we can be simultaneously right and wrong: expressing a biased position on a superficial level while holding an accurate position on a deeper level. The specifics are different in each case, but perhaps the same general mechanism may underlie both. How many other biases use this same mechanism?

Footnotes

1: In most studies on this effect, it's most commonly observed among males. The reasons are too complicated and controversial to be discussed in this post, but are left as an exercise for the reader with a background in evolutionary psychology.

2: Compare the ideal Bayesian, for whom expected future expectation is always the same as the current expectation, and investors in an ideal stock market, who must always expect a stock's price tomorrow to be on average the same as its price today - to this poor creature, who accurately predicts that he will lower his estimate of his intelligence after taking the test, but who doesn't use that prediction to change his pre-test estimates.

3: I have seen "overconfidence bias" used in two different ways: to mean poor calibration on guesses (ie predictions made with 99% certainty that are only right 70% of the time) and to mean the tendency to overestimate one's own good qualities and chance of success. I am using the latter definition here to remain consistent with the common usage on Overcoming Bias; other people may call this same error "optimism bias".

Comment author: Yvain 06 March 2009 12:21:50PM 7 points [-]

Some of the commenters on this thread seem to be viewing the change caused by psilocylbin as foreign or hostile in some way. Maybe I'm approaching it from a different perspective. My emotional tendencies fluctuate from hour to hour, day to day, and month to month. Just eating a good meal brings me from grouchy and irritable, to happy and creative. I don't feel like my identity is lost during any of these transformations.

Going from sad to happy doesn't change my utility function/preference ordering, my beliefs about the world, or my level of rationality. It just makes me able to enjoy all these things more fully. So if I were assured that the psilocybin would just change my happiness set point it would seem like a pretty good deal.

Applying the wirehead argument here seems a case of lost purposes. The problem with wireheading isn't that you're happier than you "should" be, it's that you've lost the chance to get all of those things in your utility function that aren't just happiness, because you're too busy blissing out. I see no reason to believe the psilocybin takers are at even the slightest risk of that.

A question to anyone who wouldn't take the psilocybin: if you had depression, would you take Prozac (assume for this hypothetical that Prozac is proven to work)? Do you think the answer to this question should always be the same as the psilocybin question?

Comment author: Eliezer_Yudkowsky 05 March 2009 06:58:59PM 4 points [-]

Well, yes, there is a 100% selection bias here. I'm not sure I can count that as evidence, like, at all.

Comment author: Yvain 06 March 2009 09:52:19AM 10 points [-]

The intention was to provide a clarifying example of an existential statement that should be non-controversial ("There exist some people who are uncomfortable living a lie"), not to assert probabilistic evidence for a universal statement ("Everyone I have read about is uncomfortable living a lie, therefore this is true of all humans"). I noted the selection bias only to clarify that I am not making the stronger universal statement, but it doesn't interfere with the existential statement.

Comment author: Eliezer_Yudkowsky 05 March 2009 03:47:05AM 22 points [-]

That's not a benefit of believing in God. You don't have to believe in God to be accepted into religious communities. You just have to say "I believe in God".

It may help to genuinely believe you believe in God. But in the Modern Orthodox Jewish community that I remember from Chicago, someone who actually seriously believed in God and acted accordingly, who was over the age of 20, would probably get looked at a little funny - they wouldn't get the warm friendship that accrues to those who just say the passwords.

A "benefit" of actually believing in God would be, say, that you weren't too sad at funerals because you genuinely believed the deceased was in Heaven. Pretty sure no one at the family funerals I attended went that far.

Comment author: Yvain 05 March 2009 06:21:13PM 7 points [-]

Doesn't she receive a benefit by not having to live a lie her whole life? I've read deconversion stories, and they almost always include a point where someone has lost faith but tries to stay in their religious communities and go through the motions. Most of them end up miserable (granted that there is a 100% selection bias because these are deconversion stories)

Comment author: Yvain 05 March 2009 06:17:05PM 23 points [-]

When I first read "Belief in Belief", I liked it, and agreed with it, but I thought it was describing a curiousity; an exotic specimen of irrationality for us to oooh and aaah over. I mentally applied it to Unitarians and Reform Jews and that was about it.

I've since started wondering more and more if it actually describes a majority of religious people. I don't know if this is how Eliezer intended it, but it was two things that really convinced me:

The first reason was behavior. Most theists I know occasionally deviate from their religious principles; not egregiously, but they're far from perfect. But when I imagine a world that would make me believe religion with certainty - a world where angels routinely descend to people's bedsides to carry their souls to Heaven, or where Satan allows National Geographic into Hell to film a documentary - I find it hard to imagine people sleeping in on Sundays. Not even the most hardened criminal will steal when the policeman's right in front of him and the punishment is infinite.

The second was a webcomic: http://www.heavingdeadcats.com/wp-content/uploads/2009/02/file1126-2.jpg It wasn't so much that theists wouldn't drink the poison as that they'd be surprised, even offended at being asked. It would seem like a cheap trick. Whereas (for example) I would be happy to prove my "faith" in science by ingesting poison after I'd taken an antidote proven to work in clinical trials.

I see two ways this issue is directly important to rationalists:

  1. Is this solely a religious phenomenon, or are our own beliefs vulnerable to this kind of self-deception?

  2. What kind of tests can we create to determine whether a belief is sincerely held?

Comment author: Jack 05 March 2009 10:06:31AM 4 points [-]

Occam's Razor is a heuristic... and one I proceed according to- but its not at all clear just what its justification is. Why exactly ought we to believe the simpler hypothesis?

Comment author: Yvain 05 March 2009 12:17:49PM 17 points [-]

The best justification I've heard for believing simple hypotheses is an argument from probability.

Consider some event caused by a certain block. We know the block's color must be either red, yellow, blue, or green; its shape must be either square, round, or triangular; its material must be either wood or metal.

We come up with two theories about the event. Both theories explain the event adequately:

The event was caused by the block being made of wood. The event was caused by the block being blue, and triangular, and and made of metal. Before the event happens, there are twenty four different possibile configurations of the block. "Made of wood" is true of twelve configurations, "blue, triangular, and made of metal" is true of one configuration.

After the event, we dismiss all configurations except these thirteen under which we believe the event was possible. We assume all of these thirteen are equally likely. Therefore, there's a 12/13 chance that the block is made of wood and a 1/13 chance the block is blue, triangular, and made of metal.

Therefore, Theory 1 is twelve times more likely than Theory 2.

The same principle is at work any time you have a simple theory competing with a more complex theory. Because the complicated theory has more preconditions that have to be just right, it has a lower prior probability relative to the simple theory, and since the occurence of the event adjusts the probabilities of both theories equally, it has a lower posterior probability.

I know I read this explanation first on a discussion of Kolmogorov complexity on someone's rationality blog, but I can't remember who's or what the link was. If I stole your explanation, please step up and take credit.

Comment author: Johnicholas 04 March 2009 07:16:03PM *  8 points [-]

We're talking about at least two different notions of the word "rational":

  1. Robin Hanson used the definition at the top of this post, regarding believing the truth. There are social/evolutionary costs to that, partly because humans lie poorly.

  2. The causal decision theorists' definition that Eliezer Yudkowsky was annoyed by. CDT defines rationality to be a specific method of deciding what action to take, even though this leads to two-boxing (losing) Newcomb's problem. Yudkowsky's objection, summarized by the slogan "Rationalists should WIN." was NOT a definition. It is a quality of his informal concept of rationality which the CDT definition failed to capture.

The claim "rationalists should always win" comes from taking Yudkowsky's slogan as a definition of rationality. If that is the definition that you are using, then the claim is tautological.

Please note that I don't endorse this misreading of Yudkowsky's post, I'm just trying to answer your question.

Comment author: Yvain 04 March 2009 11:01:20PM 9 points [-]

Thanks, John.

As you say, defining rationality as winning and then saying rationalists always win is a tautology. But aside from your two definitions, there's a third definition: the common definition of rationality as basing decisions on evidence, Bayes, and logic. So as I see it, supporters of "rationalists always win" need to do one of the following:

  1. Show that the winning definition is the same as the Bayes/logic/evidence definition. Tim's counterexample of the religious believer who's a poor liar makes me doubt this is possible.

  2. Stop using "rationality" to refer to things like the Twelve Virtues and Bayesian techniques, since these virtues and techniques sometimes lose and are therefore not always rational.

  3. Abandon "rationalists always win" in favor of Robin's "rationalists always seek the truth". I think that definition is sufficient to demonstrate that a rationalist should one-box on Newcombe's problem anyway. After all, if it's true that one boxing is the better result, a seeker of truth should realize that and decide to one-box.

Comment author: timtyler 04 March 2009 02:17:51PM 4 points [-]

Well, sure. Repeating other posts - but one of the most common examples is when an agent's beliefs are displayed to other agents. Imagine that all your associates think that there is a Christian god. This group includes all your prospective friends and mates. Do you tell them you are an agnostic/atheist - and that their views are not supported by the evidence? No, of course not! However, you had better not lie to them either - since most humans lie so poorly. The best thing to do is probably to believe their nonsense yourself.

Comment author: Yvain 04 March 2009 04:34:03PM 6 points [-]

Tim, that's an excellent argument for why rationality isn't always the winning strategy in real life. People have been saying this sort of thing all week, but it was your "most humans lie so poorly" comment that really made it click for me, especially in the context of evolutionary psychology.

I'd really like to hear one of the "rationalists should always win" people address this objection.

Comment author: Yvain 04 March 2009 11:45:42AM 13 points [-]

I am likely to buy the book for my friends in any case, but I did just think of something that would give me an extra incentive. I want my friends to understand consequentialism/utility.

Most people I know are very stuck in "morality means not doing anything bad" mode. Not only would they not push the fat person off the bridge in the trolley problem, but many (perhaps most) wouldn't so much as push the lever to switch tracks. A lot of them think the whole question is some sort of reductio ad absurdum of consequentialist philosophy ("you're a consequentialist? I thought that was discredited when people showed that philosophy could be used to justify murdering someone in that trolley problem"). I've had no success arguing against these people, and one of the things that impressed me most the first time I read Overcoming Bias was that it was a community of people who used consequentialist arguments without fumbling for excuses, who thought Torture Vs. Dust Specks was a real problem instead of just "It would be wrong to inflict either of those options on people, so it's not a moral issue" (which is what my old philosophy professor said when I mentioned it to her).

I don't think Overcoming Bias ever officially argued for consequentialism, but a book aimed at the general public might have to. Eliezer is one of the most persuasive writers I've ever read, and if I thought his book could convince my friends to think consequentially I would be recommending it even more than I would otherwise.

Comment author: Yvain 28 February 2009 09:25:39PM 39 points [-]

Since it would be impossible to disentangle and explain all the different factors, and all the studies say people are terrible at determining what events influence them anyway, I'll just tell the event in my transition to rationalism that makes for the best story:

When I was around five, my kindergarten teacher decided to initiate my friends and I into the Great Miracle Of Life by bringing an incubator full of chicken eggs into the classroom. I watched them hatch, loved the little chicks, and (after some time and other events including a bad experience at a meat-filled Asian restaurant) decided to become a vegetarian and eat neither meat nor eggs.

When I was about eleven, I got quite into politics, and like most people in my area ended up as a typical liberal. And so I was of course pro-choice: why should we respect the rights of fetuses when they're just a collection of cells and not even really alive?

It took me a while to realize that I was simultaneously refusing to eat eggs because potential-chickens were valuable living beings who deserved respect, and condoning abortion because potential-humans weren't.

If I'd been a little older and a little cleverer, I'd have made up some typical political excuse why it was really about freedom or human rights or society or something (or else just learned the difference between fertilized and unfertilized eggs!)

But I was young and innocent enough to take a moment to think "Maybe my brain is just telling me what it thinks I want to hear in each situation, instead of really thinking things through. I should find a way to stop that."

Comment author: Yvain 28 February 2009 08:57:55PM 3 points [-]

How to take Joy In The Merely Real.

Comment author: Yvain 28 February 2009 04:09:34PM 11 points [-]

The most important thing I learned from Overcoming Bias was to stop viewing the human mind as a blank slate, ideally a blank slate, an approximation to a blank slate, or anything with properties even slightly resembling blankness or slateness. The rest is just commentary - admittedly very, very good commentary.

The posts I associate with this are everything on evolutionary psychology such as Godshatter (second most important thing I learned: study evolutionary psychology!), the free will series, the "ghost in the machine" and "ideal philosopher of perfect emptiness" series, and the Mind Projection Fallacy.